Model reports

ServerBench

编程智能体能否做出正确、高效的模型服务，并选好并发负载？ServerBench 给它八小时开发，再用新的会话回放检验服务价值。

原论文官方仓库官方项目收录来源

AI Benchmark Radar / Equality Machine · 核对于 2026-11-05

2026 年 11 月 4 日提交的原论文 v1；范围为三种 L40 部署与表 1。11 月 5 日核对，实验日期未明确。

既开发服务，也选择负载

智能体在离线 Harbor 环境中调整批处理、调度、缓存与内核。AgentX 回放保留轮次依赖、共享前缀与并行分支；内容经过重构，但长度和依赖保持。三个模型／硬件任务分别使用不同价格与延迟门槛。

ServerBench · original paper v1

每美元 GPU 成本对应多少合格服务价值

只有完成且同时满足首 token 延迟和逐 token 延迟门槛的请求，其 token 价值才计入分子。分母为 GPU 配额与测量时长对应成本。缓存输入按每个逻辑请求完整计价一次。开发阶段 API 花费另算；该比值不是利润或百分比。

ServerBench · original paper v1

通过验证才有成绩

每个智能体／部署组合有八小时开发和 1,800 秒最终回放。使用最高推理档位；GPT 通过 Codex，其他智能体通过 Claude Code。对异常失败最多重跑一次，报告最强的已验证最终提交。评分前重新部署，使用不重叠种子，并检查保真度、执行完整性与代码。

ServerBench · original paper v1

Gemma-4-E4B-it / 1×L40

原表 1。验证未通过的组合没有数值成绩；33 个组合中有 22 个通过。首 token 延迟单位为秒，逐 token 延迟单位为毫秒。

表 1 · Gemma-4-E4B-it / 1×L40
智能体	美元／美元	并发会话数	p90 首 token（秒）	p90 逐 token（毫秒）
Claude Opus 5.5	2.47	40	3.01	118.59
Claude Sonnet 5.5	2.97	32	2.75	83.05
GPT 6 Astra	1.75	32	4.77	94.19
GPT 6.1 Sol	1.59	24	4.43	72.45
Claude Opus 5	1.15	24	3.98	109.12
GPT 6 Sol	0.55	5	3.63	110.52
Kimi K3	0.52	6	8.67	55.32
Qwen3.8-Max-0902	0.87	14	2.61	59.16
MiMo V2.6 Pro	0.22	10	43.63	108.96
GLM 5.3	Not passed	—	—	—
DeepSeek V4.1 Flash	Not passed	—	—	—

Not passed 保留未通过验证且无数值成绩的状态，不是零分。

ServerBench · original paper v1

Qwen3.5-9B / 2×L40

原表 1。验证未通过的组合没有数值成绩；33 个组合中有 22 个通过。首 token 延迟单位为秒，逐 token 延迟单位为毫秒。

表 1 · Qwen3.5-9B / 2×L40
智能体	美元／美元	并发会话数	p90 首 token（秒）	p90 逐 token（毫秒）
Claude Opus 5.5	4.40	16	2.25	28.16
Claude Sonnet 5.5	4.34	20	12.34	26.76
GPT 6 Astra	3.18	12	15.62	21.94
GPT 6.1 Sol	2.84	4	2.71	23.67
Claude Opus 5	1.16	1	5.51	28.09
GPT 6 Sol	1.11	1	13.19	20.45
Kimi K3	0.93	1	4.99	33.12
Qwen3.8-Max-0902	Not passed	—	—	—
MiMo V2.6 Pro	Not passed	—	—	—
GLM 5.3	Not passed	—	—	—
DeepSeek V4.1 Flash	Not passed	—	—	—

Not passed 保留未通过验证且无数值成绩的状态，不是零分。

ServerBench · original paper v1

Qwen3.8-27B-INT4 / 2×L40

原表 1。验证未通过的组合没有数值成绩；33 个组合中有 22 个通过。首 token 延迟单位为秒，逐 token 延迟单位为毫秒。

表 1 · Qwen3.8-27B-INT4 / 2×L40
智能体	美元／美元	并发会话数	p90 首 token（秒）	p90 逐 token（毫秒）
Claude Opus 5.5	2.64	8	11.44	46.40
Claude Sonnet 5.5	2.01	12	46.57	47.35
GPT 6 Astra	1.48	4	12.96	42.83
GPT 6.1 Sol	0.98	4	75.66	43.31
Claude Opus 5	0.62	6	33.55	62.97
GPT 6 Sol	0.26	2	136.48	49.53
Kimi K3	Not passed	—	—	—
Qwen3.8-Max-0902	Not passed	—	—	—
MiMo V2.6 Pro	Not passed	—	—	—
GLM 5.3	Not passed	—	—	—
DeepSeek V4.1 Flash	Not passed	—	—	—

Not passed 保留未通过验证且无数值成绩的状态，不是零分。

ServerBench · original paper v1

框架参考系统

参考成绩是在固定并发下，对五个独立回放种子取平均。它们是框架部署，与编程智能体提交分别展示。

表 1 · 框架参考值
部署	框架	美元／美元	并发会话数
Gemma E4B / 1×L40	vLLM v0.28.0	0.61	8
Gemma E4B / 1×L40	SGLang v0.5.19	0.16	2
Qwen3.5 9B / 2×L40	SGLang v0.5.19	1.62	1
Qwen3.5 9B / 2×L40	vLLM v0.28.0	1.49	1
Qwen3.8 27B INT4 / 2×L40	vLLM v0.28.0	1.47	1
Qwen3.8 27B INT4 / 2×L40	SGLang v0.5.19	1.42	4

框架参考值取五个种子平均，不是智能体开发运行。

ServerBench · original paper v1

冻结的部署参数

改变检查点、硬件、负载或推理策略需要重新校准。这里记录论文设置，不代表当前商业报价。

表 2 · 部署参数
参数	Qwen3.5 9B	Gemma E4B	Qwen3.8 27B INT4
GPU allocation	2×L40	1×L40	2×L40
Context limit (tokens)	1,048,576	131,072	1,048,576
Input price (USD / million tokens)	0.1044	0.0200	0.0898
Output price (USD / million tokens)	0.1855	0.1000	2.6570
TTFT target (s)	4.10028	2.52693	10.2988
TPOT target (ms)	23.4974	121.7057	48.3769
GPU cost (USD / GPU-hour)	0.82	0.82	0.82
SGLang reference concurrency	1	2	4
vLLM reference concurrency	1	8	1

论文冻结的价格与延迟门槛；硬件列顺序按原表 2。

ServerBench · original paper v1

开发花费单列

这些 API 花费使用 2026 年 10 月 13 日获取的 OpenRouter 费率，对记录中的 token 与缓存用量重新计价，不替代服务成绩中 GPU 成本的分母。

表 3 · 开发 API 花费估算（美元）
智能体	Qwen3.5 9B	Gemma E4B
Claude Opus 5.5	99.69	104.25
Claude Sonnet 5.5	78.60	30.72
GPT 6 Astra	110.81	119.79
GPT 6.1 Sol	13.01	13.67
Claude Opus 5	108.90	104.37
GPT 6 Sol	50.88	35.93
Kimi K3	126.98	28.71
Qwen3.8-Max-0902	46.13	43.33
MiMo V2.6 Pro	6.32	10.19
GLM 5.3	16.80	10.79
DeepSeek V4.1 Flash	1.64	2.07

美元估算使用 2026 年 10 月 13 日获取的费率，与服务 GPU 成本分别记录。

ServerBench · original paper v1

固定负载下的版本对比

这些比值在匹配的回放种子下比较连续实现版本，不作为主表的新模型。

表 4 · 连续版本服务价值比值
系统	版本	固定并发	匹配种子下比值
Opus 5.5 / Qwen3.5 9B	v1 → v2	16	1.88, 1.32
Opus 5 / Qwen3.5 9B	v3 → v4	2	1.90, 1.67, 1.44
Sonnet 5.5 / Gemma E4B	v3 → v4	32	1.24, 1.02, 1.05

每个比值对应给定固定负载下的一对可评分匹配种子实验。

ServerBench · original paper v1

受控后续实验

这些冻结服务器的后续实验使用新的公共种子。容量对比是在并发 16 下将 Qwen v1 改为 v6；草稿长度对比在并发 40 下将 Gemma 的 K=3 改为 K=7；回放反馈对比两台恢复运行的 Opus 5 Gemma 服务器。D 为合格价值，G 为总价值，f 为合格占比。所有实验臂均通过验证。

表 5 · 匹配种子后续实验
对比	种子	D before	D after	D ratio	G ratio	f ratio
Capacity	D	3.533	4.761	1.348	1.110	1.214
Capacity	E	4.294	4.960	1.155	1.074	1.076
Draft length	D	2.240	2.308	1.030	1.030	1.000
Draft length	E	2.450	2.544	1.039	1.029	1.009
Replay feedback	F	0.786	0.888	1.129	1.112	1.015
Replay feedback	G	0.878	1.148	1.308	1.243	1.052

前后比值描述受控服务器变体，与主表的智能体对比分别展示。

ServerBench · original paper v1

按具体部署阅读成绩

成绩取决于计价、延迟门槛、并发负载与验证条件。对冻结实现反复回放，不等于独立运行智能体重新构建。论文发布日期已知；具体实验日期及完整服务端模型版本需查实验记录。

ServerBench · original paper v1

原始发布

2026 年 11 月 4 日提交的原论文 v1；范围为三种 L40 部署与表 1。11 月 5 日核对，实验日期未明确。

核对于 2026-11-05

已收录的评测资料

这条来源记录收录了 22 条数值成绩。来源区分了 9 个模型身份或配置。同一模型可能有多条记录；比较前请核对具体版本、工具和运行预算。查看成绩及逐条证据 →

报告指标: latency-qualified token value per GPU dollar
原始单位: ratio
指标方向: 数值越高越好

从这些资料开始

收录来源 · arxiv.org

引用文档及日期（1）

ServerBench v1 · original paper results · 2026-11-04

评测信息与证据

来源: Model reports
发布者: 未收录
评测发布日期: 2026-11-04
采集日期: 未收录
模态: 未收录

本站保留来源报告的结果，不代表独立复测。请先核对任务版本、评测协议、分数单位和报告日期；不同运行条件不能直接合并为统一排名。

报告成绩 22

按来源中的观察顺序展示，顺序不代表排名。日期保留原始精度。

模型	数值	单位	协议	报告日期与口径	来源
MiMo V2.6 Pro	0.22	ratio	Paper Table 1; Gemma-4-E4B-it / 1×L40; concurrency 10; maximum reasoning effort; Claude Code in Harbor; 8-hour development, 1800-second final replay, held-out seeds; strongest verified submission with at most one anomalous-failure rerun; TTFT ≤ 2.52693 s, TPOT ≤ 121.7057 ms; allocated GPU price $0.82/GPU-hour; paper-fixed input/output prices and full cached-prefix valuation; actual run dates and complete provider revision unspecified	2026-11-04 · document_publication	ServerBench v1 · original paper results
Qwen3.8-Max-0902	0.87	ratio	Paper Table 1; Gemma-4-E4B-it / 1×L40; concurrency 14; maximum reasoning effort; Claude Code in Harbor; 8-hour development, 1800-second final replay, held-out seeds; strongest verified submission with at most one anomalous-failure rerun; TTFT ≤ 2.52693 s, TPOT ≤ 121.7057 ms; allocated GPU price $0.82/GPU-hour; paper-fixed input/output prices and full cached-prefix valuation; actual run dates and complete provider revision unspecified	2026-11-04 · document_publication	ServerBench v1 · original paper results
Claude Opus 5	1.15	ratio	Paper Table 1; Gemma-4-E4B-it / 1×L40; concurrency 24; maximum reasoning effort; Claude Code in Harbor; 8-hour development, 1800-second final replay, held-out seeds; strongest verified submission with at most one anomalous-failure rerun; TTFT ≤ 2.52693 s, TPOT ≤ 121.7057 ms; allocated GPU price $0.82/GPU-hour; paper-fixed input/output prices and full cached-prefix valuation; actual run dates and complete provider revision unspecified	2026-11-04 · document_publication	ServerBench v1 · original paper results
GPT 6.1 Sol	1.59	ratio	Paper Table 1; Gemma-4-E4B-it / 1×L40; concurrency 24; maximum reasoning effort; Codex in Harbor; 8-hour development, 1800-second final replay, held-out seeds; strongest verified submission with at most one anomalous-failure rerun; TTFT ≤ 2.52693 s, TPOT ≤ 121.7057 ms; allocated GPU price $0.82/GPU-hour; paper-fixed input/output prices and full cached-prefix valuation; actual run dates and complete provider revision unspecified	2026-11-04 · document_publication	ServerBench v1 · original paper results
Claude Sonnet 5.5	2.97	ratio	Paper Table 1; Gemma-4-E4B-it / 1×L40; concurrency 32; maximum reasoning effort; Claude Code in Harbor; 8-hour development, 1800-second final replay, held-out seeds; strongest verified submission with at most one anomalous-failure rerun; TTFT ≤ 2.52693 s, TPOT ≤ 121.7057 ms; allocated GPU price $0.82/GPU-hour; paper-fixed input/output prices and full cached-prefix valuation; actual run dates and complete provider revision unspecified	2026-11-04 · document_publication	ServerBench v1 · original paper results
GPT 6 Astra	1.75	ratio	Paper Table 1; Gemma-4-E4B-it / 1×L40; concurrency 32; maximum reasoning effort; Codex in Harbor; 8-hour development, 1800-second final replay, held-out seeds; strongest verified submission with at most one anomalous-failure rerun; TTFT ≤ 2.52693 s, TPOT ≤ 121.7057 ms; allocated GPU price $0.82/GPU-hour; paper-fixed input/output prices and full cached-prefix valuation; actual run dates and complete provider revision unspecified	2026-11-04 · document_publication	ServerBench v1 · original paper results
Claude Opus 5.5	2.47	ratio	Paper Table 1; Gemma-4-E4B-it / 1×L40; concurrency 40; maximum reasoning effort; Claude Code in Harbor; 8-hour development, 1800-second final replay, held-out seeds; strongest verified submission with at most one anomalous-failure rerun; TTFT ≤ 2.52693 s, TPOT ≤ 121.7057 ms; allocated GPU price $0.82/GPU-hour; paper-fixed input/output prices and full cached-prefix valuation; actual run dates and complete provider revision unspecified	2026-11-04 · document_publication	ServerBench v1 · original paper results
GPT 6 Sol	0.55	ratio	Paper Table 1; Gemma-4-E4B-it / 1×L40; concurrency 5; maximum reasoning effort; Codex in Harbor; 8-hour development, 1800-second final replay, held-out seeds; strongest verified submission with at most one anomalous-failure rerun; TTFT ≤ 2.52693 s, TPOT ≤ 121.7057 ms; allocated GPU price $0.82/GPU-hour; paper-fixed input/output prices and full cached-prefix valuation; actual run dates and complete provider revision unspecified	2026-11-04 · document_publication	ServerBench v1 · original paper results
Kimi K3	0.52	ratio	Paper Table 1; Gemma-4-E4B-it / 1×L40; concurrency 6; maximum reasoning effort; Claude Code in Harbor; 8-hour development, 1800-second final replay, held-out seeds; strongest verified submission with at most one anomalous-failure rerun; TTFT ≤ 2.52693 s, TPOT ≤ 121.7057 ms; allocated GPU price $0.82/GPU-hour; paper-fixed input/output prices and full cached-prefix valuation; actual run dates and complete provider revision unspecified	2026-11-04 · document_publication	ServerBench v1 · original paper results
GPT 6 Astra	3.18	ratio	Paper Table 1; Qwen3.5-9B / 2×L40; concurrency 12; maximum reasoning effort; Codex in Harbor; 8-hour development, 1800-second final replay, held-out seeds; strongest verified submission with at most one anomalous-failure rerun; TTFT ≤ 4.10028 s, TPOT ≤ 23.4974 ms; allocated GPU price $0.82/GPU-hour; paper-fixed input/output prices and full cached-prefix valuation; actual run dates and complete provider revision unspecified	2026-11-04 · document_publication	ServerBench v1 · original paper results
Claude Opus 5.5	4.4	ratio	Paper Table 1; Qwen3.5-9B / 2×L40; concurrency 16; maximum reasoning effort; Claude Code in Harbor; 8-hour development, 1800-second final replay, held-out seeds; strongest verified submission with at most one anomalous-failure rerun; TTFT ≤ 4.10028 s, TPOT ≤ 23.4974 ms; allocated GPU price $0.82/GPU-hour; paper-fixed input/output prices and full cached-prefix valuation; actual run dates and complete provider revision unspecified	2026-11-04 · document_publication	ServerBench v1 · original paper results
Claude Opus 5	1.16	ratio	Paper Table 1; Qwen3.5-9B / 2×L40; concurrency 1; maximum reasoning effort; Claude Code in Harbor; 8-hour development, 1800-second final replay, held-out seeds; strongest verified submission with at most one anomalous-failure rerun; TTFT ≤ 4.10028 s, TPOT ≤ 23.4974 ms; allocated GPU price $0.82/GPU-hour; paper-fixed input/output prices and full cached-prefix valuation; actual run dates and complete provider revision unspecified	2026-11-04 · document_publication	ServerBench v1 · original paper results
Kimi K3	0.93	ratio	Paper Table 1; Qwen3.5-9B / 2×L40; concurrency 1; maximum reasoning effort; Claude Code in Harbor; 8-hour development, 1800-second final replay, held-out seeds; strongest verified submission with at most one anomalous-failure rerun; TTFT ≤ 4.10028 s, TPOT ≤ 23.4974 ms; allocated GPU price $0.82/GPU-hour; paper-fixed input/output prices and full cached-prefix valuation; actual run dates and complete provider revision unspecified	2026-11-04 · document_publication	ServerBench v1 · original paper results
GPT 6 Sol	1.11	ratio	Paper Table 1; Qwen3.5-9B / 2×L40; concurrency 1; maximum reasoning effort; Codex in Harbor; 8-hour development, 1800-second final replay, held-out seeds; strongest verified submission with at most one anomalous-failure rerun; TTFT ≤ 4.10028 s, TPOT ≤ 23.4974 ms; allocated GPU price $0.82/GPU-hour; paper-fixed input/output prices and full cached-prefix valuation; actual run dates and complete provider revision unspecified	2026-11-04 · document_publication	ServerBench v1 · original paper results
Claude Sonnet 5.5	4.34	ratio	Paper Table 1; Qwen3.5-9B / 2×L40; concurrency 20; maximum reasoning effort; Claude Code in Harbor; 8-hour development, 1800-second final replay, held-out seeds; strongest verified submission with at most one anomalous-failure rerun; TTFT ≤ 4.10028 s, TPOT ≤ 23.4974 ms; allocated GPU price $0.82/GPU-hour; paper-fixed input/output prices and full cached-prefix valuation; actual run dates and complete provider revision unspecified	2026-11-04 · document_publication	ServerBench v1 · original paper results
GPT 6.1 Sol	2.84	ratio	Paper Table 1; Qwen3.5-9B / 2×L40; concurrency 4; maximum reasoning effort; Codex in Harbor; 8-hour development, 1800-second final replay, held-out seeds; strongest verified submission with at most one anomalous-failure rerun; TTFT ≤ 4.10028 s, TPOT ≤ 23.4974 ms; allocated GPU price $0.82/GPU-hour; paper-fixed input/output prices and full cached-prefix valuation; actual run dates and complete provider revision unspecified	2026-11-04 · document_publication	ServerBench v1 · original paper results
Claude Sonnet 5.5	2.01	ratio	Paper Table 1; Qwen3.8-27B-INT4 / 2×L40; concurrency 12; maximum reasoning effort; Claude Code in Harbor; 8-hour development, 1800-second final replay, held-out seeds; strongest verified submission with at most one anomalous-failure rerun; TTFT ≤ 10.2988 s, TPOT ≤ 48.3769 ms; allocated GPU price $0.82/GPU-hour; paper-fixed input/output prices and full cached-prefix valuation; actual run dates and complete provider revision unspecified	2026-11-04 · document_publication	ServerBench v1 · original paper results
GPT 6 Sol	0.26	ratio	Paper Table 1; Qwen3.8-27B-INT4 / 2×L40; concurrency 2; maximum reasoning effort; Codex in Harbor; 8-hour development, 1800-second final replay, held-out seeds; strongest verified submission with at most one anomalous-failure rerun; TTFT ≤ 10.2988 s, TPOT ≤ 48.3769 ms; allocated GPU price $0.82/GPU-hour; paper-fixed input/output prices and full cached-prefix valuation; actual run dates and complete provider revision unspecified	2026-11-04 · document_publication	ServerBench v1 · original paper results
GPT 6 Astra	1.48	ratio	Paper Table 1; Qwen3.8-27B-INT4 / 2×L40; concurrency 4; maximum reasoning effort; Codex in Harbor; 8-hour development, 1800-second final replay, held-out seeds; strongest verified submission with at most one anomalous-failure rerun; TTFT ≤ 10.2988 s, TPOT ≤ 48.3769 ms; allocated GPU price $0.82/GPU-hour; paper-fixed input/output prices and full cached-prefix valuation; actual run dates and complete provider revision unspecified	2026-11-04 · document_publication	ServerBench v1 · original paper results
GPT 6.1 Sol	0.98	ratio	Paper Table 1; Qwen3.8-27B-INT4 / 2×L40; concurrency 4; maximum reasoning effort; Codex in Harbor; 8-hour development, 1800-second final replay, held-out seeds; strongest verified submission with at most one anomalous-failure rerun; TTFT ≤ 10.2988 s, TPOT ≤ 48.3769 ms; allocated GPU price $0.82/GPU-hour; paper-fixed input/output prices and full cached-prefix valuation; actual run dates and complete provider revision unspecified	2026-11-04 · document_publication	ServerBench v1 · original paper results
Claude Opus 5	0.62	ratio	Paper Table 1; Qwen3.8-27B-INT4 / 2×L40; concurrency 6; maximum reasoning effort; Claude Code in Harbor; 8-hour development, 1800-second final replay, held-out seeds; strongest verified submission with at most one anomalous-failure rerun; TTFT ≤ 10.2988 s, TPOT ≤ 48.3769 ms; allocated GPU price $0.82/GPU-hour; paper-fixed input/output prices and full cached-prefix valuation; actual run dates and complete provider revision unspecified	2026-11-04 · document_publication	ServerBench v1 · original paper results
Claude Opus 5.5	2.64	ratio	Paper Table 1; Qwen3.8-27B-INT4 / 2×L40; concurrency 8; maximum reasoning effort; Claude Code in Harbor; 8-hour development, 1800-second final replay, held-out seeds; strongest verified submission with at most one anomalous-failure rerun; TTFT ≤ 10.2988 s, TPOT ≤ 48.3769 ms; allocated GPU price $0.82/GPU-hour; paper-fixed input/output prices and full cached-prefix valuation; actual run dates and complete provider revision unspecified	2026-11-04 · document_publication	ServerBench v1 · original paper results

既开发服务，也选择负载

每美元 GPU 成本对应多少合格服务价值

通过验证才有成绩

Gemma-4-E4B-it / 1×L40

Qwen3.5-9B / 2×L40

Qwen3.8-27B-INT4 / 2×L40

框架参考系统

冻结的部署参数

开发花费单列

固定负载下的版本对比

受控后续实验

按具体部署阅读成绩

原始发布

已收录的评测资料

从这些资料开始

评测信息与证据

报告成绩 22

相关评测探索

继续阅读

继续探索