ARCBench — Model reports · AI Benchmark Radar

AI Benchmark Radar / Equality Machine · 核对于 2026-11-04

2026 年 11 月 2 日原论文 v1，11 月 4 日核查。主成绩为 S1 基线，其余模型、渠道、策略、预算与攻防实验独立标注。

压缩历史后的规则

17 个测试场景覆盖泄露数据、越权访问、未授权操作与必要控制；在 pi Agent 框架中，根据实际工具调用判断违规。

ARCBench · original paper v1

越低越好；题例不等于运行

violation@3 统计三次有效重复中至少违规一次的配置。S1 保留 135 个配置，排除 GPT-Luna 的六次无效运行；完整与压缩历史成对比较。

ARCBench · original paper v1

保留模型端点与框架设置

执行器与压缩器使用同一端点。S1 使用 Hermes，无攻击和防御，温度 0.7，最多四次工具操作；实际运行日期未知。

ARCBench · original paper v1

保留规则是另一项测量

规则状态与行为解释由独立裁判标注。模拟工具与较短场景限制泛化；violation@6 和逐运行违规率不能当作 violation@3 比较。

ARCBench · original paper v1

S1：完整历史与 Hermes

题例级违规与运行级规则状态的分母不同。

S1 原文基线
配置	完整历史 %	已有违规 %	仅压缩后 %	压缩后总计 %	规则完整 %	规则部分 %	规则缺失 %
All	9.6	8.9	28.9	37.8	70.1	26.9	3.0
DeepSeek	8.8	8.8	17.6	26.5	98.0	2.0	0.0
GPT-Luna	6.1	6.1	27.3	33.3	48.5	45.5	6.1
Qwen	14.7	11.8	50.0	61.8	35.3	58.8	5.9
GLM	8.8	8.8	20.6	29.4	98.0	2.0	0.0

Full/Own/After/Total 为 violation@3 题例占比；末三列统计压缩后运行的规则状态。

ARCBench original results · Table 3

S1 模型与规则渠道

请按该表配置独立阅读，与主成绩 S1 分开比较。

S1 模型与规则渠道
分组	配置	题例数	完整历史 %	已有违规 %	仅压缩后 %	压缩后总计 %	规则完整 %	规则部分 %	规则缺失 %	仅完整历史 %
All	All	135	9.6 (13)	8.9 (12)	28.9 (39)	37.8 (51)	70.1	26.9	3.0	0.7 (1)
All	c1	32	25.0 (8)	25.0 (8)	31.3 (10)	56.3 (18)	74.0	24.0	2.1	0.0 (0)
All	c2	31	16.1 (5)	12.9 (4)	32.3 (10)	45.2 (14)	69.9	25.8	4.3	3.2 (1)
All	c3	36	0.0 (0)	0.0 (0)	30.6 (11)	30.6 (11)	65.7	29.6	4.6	0.0 (0)
All	c4	36	0.0 (0)	0.0 (0)	22.2 (8)	22.2 (8)	71.3	27.8	0.9	0.0 (0)
DeepSeek	DeepSeek	34	8.8 (3)	8.8 (3)	17.6 (6)	26.5 (9)	98.0	2.0	0.0	0.0 (0)
DeepSeek	c1	8	25.0 (2)	25.0 (2)	37.5 (3)	62.5 (5)	95.8	4.2	0.0	0.0 (0)
DeepSeek	c2	8	12.5 (1)	12.5 (1)	25.0 (2)	37.5 (3)	100.0	0.0	0.0	0.0 (0)
DeepSeek	c3	9	0.0 (0)	0.0 (0)	11.1 (1)	11.1 (1)	96.3	3.7	0.0	0.0 (0)
DeepSeek	c4	9	0.0 (0)	0.0 (0)	0.0 (0)	0.0 (0)	100.0	0.0	0.0	0.0 (0)
GPT-Luna	GPT-Luna	33	6.1 (2)	6.1 (2)	27.3 (9)	33.3 (11)	48.5	45.5	6.1	0.0 (0)
GPT-Luna	c1	8	12.5 (1)	12.5 (1)	12.5 (1)	25.0 (2)	70.8	29.2	0.0	0.0 (0)
GPT-Luna	c2	7	14.3 (1)	14.3 (1)	14.3 (1)	28.6 (2)	47.6	47.6	4.8	0.0 (0)
GPT-Luna	c3	9	0.0 (0)	0.0 (0)	44.4 (4)	44.4 (4)	40.7	40.7	18.5	0.0 (0)
GPT-Luna	c4	9	0.0 (0)	0.0 (0)	33.3 (3)	33.3 (3)	37.0	63.0	0.0	0.0 (0)
Qwen	Qwen	34	14.7 (5)	11.8 (4)	50.0 (17)	61.8 (21)	35.3	58.8	5.9	2.9 (1)
Qwen	c1	8	37.5 (3)	37.5 (3)	37.5 (3)	75.0 (6)	29.2	62.5	8.3	0.0 (0)
Qwen	c2	8	25.0 (2)	12.5 (1)	50.0 (4)	62.5 (5)	29.2	58.3	12.5	12.5 (1)
Qwen	c3	9	0.0 (0)	0.0 (0)	55.6 (5)	55.6 (5)	29.6	70.4	0.0	0.0 (0)
Qwen	c4	9	0.0 (0)	0.0 (0)	55.6 (5)	55.6 (5)	51.9	44.4	3.7	0.0 (0)
GLM	GLM	34	8.8 (3)	8.8 (3)	20.6 (7)	29.4 (10)	98.0	2.0	0.0	0.0 (0)
GLM	c1	8	25.0 (2)	25.0 (2)	37.5 (3)	62.5 (5)	100.0	0.0	0.0	0.0 (0)
GLM	c2	8	12.5 (1)	12.5 (1)	37.5 (3)	50.0 (4)	100.0	0.0	0.0	0.0 (0)
GLM	c3	9	0.0 (0)	0.0 (0)	11.1 (1)	11.1 (1)	96.3	3.7	0.0	0.0 (0)
GLM	c4	9	0.0 (0)	0.0 (0)	0.0 (0)	0.0 (0)	96.3	3.7	0.0	0.0 (0)

括号为违规题例数。c1/c2 为个人规则渠道，c3/c4 为组织规则渠道；GPT-Luna 有 33 个有效配置。

ARCBench original results · Table 18

S1 逐运行结果

请按该表配置独立阅读，与主成绩 S1 分开比较。

S1 逐运行结果
分组	运行条件	占比 %	未违规 %	违规 %
All	Full context	–	91.9	8.1
All	Compacted, rule retained	70.1	78.9	21.1
All	Compacted, partially retained	26.9	70.6	29.4
All	Compacted, rule absent	3.0	16.7	83.3
DeepSeek	Full context	–	93.1	6.9
DeepSeek	Compacted, rule retained	98.0	83.0	17.0
DeepSeek	Compacted, partially retained	2.0	100.0	0.0
DeepSeek	Compacted, rule absent	0.0	–	–
GPT-Luna	Full context	–	96.0	4.0
GPT-Luna	Compacted, rule retained	48.5	79.2	20.8
GPT-Luna	Compacted, partially retained	45.5	75.6	24.4
GPT-Luna	Compacted, rule absent	6.1	33.3	66.7
Qwen	Full context	–	86.3	13.7
Qwen	Compacted, rule retained	35.3	52.8	47.2
Qwen	Compacted, partially retained	58.8	65.0	35.0
Qwen	Compacted, rule absent	5.9	0.0	100.0
GLM	Full context	–	92.2	7.8
GLM	Compacted, rule retained	98.0	84.0	16.0
GLM	Compacted, partially retained	2.0	100.0	0.0
GLM	Compacted, rule absent	0.0	–	–

这里为逐运行比例，不是 violation@3。

ARCBench original results · Table 19

S1 行为归因数量

请按该表配置独立阅读，与主成绩 S1 分开比较。

S1 行为归因数量
分组	模型	运行数 n	当作批准	范围解释	未提规则	承认仍违规	无法观察
Full context, agent’s own	All	30	21	2	5	1	1
Full context, agent’s own	DeepSeek	7	7	0	0	0	0
Full context, agent’s own	GPT-Luna	4	2	0	0	1	1
Full context, agent’s own	Qwen	11	4	2	5	0	0
Full context, agent’s own	GLM	8	8	0	0	0	0
Rule retained, agent’s own	All	20	14	1	2	0	3
Rule retained, agent’s own	DeepSeek	7	7	0	0	0	0
Rule retained, agent’s own	GPT-Luna	2	0	0	0	0	2
Rule retained, agent’s own	Qwen	2	0	1	1	0	0
Rule retained, agent’s own	GLM	9	7	0	1	0	1
Rule retained, only after compaction	All	40	17	2	10	8	3
Rule retained, only after compaction	DeepSeek	10	7	0	0	3	0
Rule retained, only after compaction	GPT-Luna	8	0	0	5	0	3
Rule retained, only after compaction	Qwen	15	6	1	4	4	0
Rule retained, only after compaction	GLM	7	4	1	1	1	0

数量，不是模型成功分数。

ARCBench original results · Table 20

S3 压缩策略比较

请按该表配置独立阅读，与主成绩 S1 分开比较。

S3 压缩策略比较
配置	题例数	完整历史 %	已有违规 %	仅压缩后 %	压缩后总计 %	规则完整 %	规则部分 %	规则缺失 %	仅完整历史 %
Generative	468	12.0 (56)	10.0 (47)	11.8 (55)	21.8 (102)	91.2	7.8	0.9	1.9 (9)
Pi + RP	67	11.9 (8)	6.0 (4)	3.0 (2)	9.0 (6)	99.0	1.0	0.0	6.0 (4)
Pi	67	11.9 (8)	9.0 (6)	6.0 (4)	14.9 (10)	96.0	4.0	0.0	3.0 (2)
Structured	67	11.9 (8)	9.0 (6)	6.0 (4)	14.9 (10)	94.5	5.0	0.5	3.0 (2)
TRACE	65	12.3 (8)	10.8 (7)	7.7 (5)	18.5 (12)	93.3	5.1	1.5	1.5 (1)
ACON-UTCO	68	11.8 (8)	11.8 (8)	8.8 (6)	20.6 (14)	90.7	8.8	0.5	0.0 (0)
ACON-UT	66	12.1 (8)	12.1 (8)	19.7 (13)	31.8 (21)	96.0	3.5	0.5	0.0 (0)
Hermes	68	11.8 (8)	11.8 (8)	30.9 (21)	42.6 (29)	69.6	27.0	3.4	0.0 (0)
Selective	203	11.8 (24)	9.9 (20)	40.4 (82)	50.2 (102)	15.9	0.0	84.1	2.0 (4)
LRU (LLM)	67	11.9 (8)	9.0 (6)	35.8 (24)	44.8 (30)	36.3	0.0	63.7	3.0 (2)
FIFO	68	11.8 (8)	10.3 (7)	41.2 (28)	51.5 (35)	0.0	0.0	100.0	1.5 (1)
LRU (rule)	68	11.8 (8)	10.3 (7)	44.1 (30)	54.4 (37)	11.8	0.0	88.2	1.5 (1)

c1/c3、无攻击、无额外预算；七种生成式与三种选择式配置。类别汇总行合并不同配置。

ARCBench original results · Table 21

S3 各模型的策略结果

请按该表配置独立阅读，与主成绩 S1 分开比较。

S3 各模型的策略结果
策略	DeepSeek 完整	DeepSeek 压缩	GPT-Luna 完整	GPT-Luna 压缩	Qwen 完整	Qwen 压缩	GLM 完整	GLM 压缩
Generative	12.2	20.0	5.9	14.3	17.6	31.1	12.2	21.7
Pi + RP	11.8	11.8	5.9	5.9	17.6	5.9	12.5	12.5
Pi	11.8	11.8	5.9	11.8	17.6	17.6	12.5	18.8
Structured	11.8	11.8	5.9	5.9	17.6	29.4	12.5	12.5
TRACE	14.3	21.4	5.9	17.6	17.6	23.5	11.8	11.8
ACON-UTCO	11.8	23.5	5.9	5.9	17.6	29.4	11.8	23.5
ACON-UT	12.5	25.0	5.9	17.6	17.6	47.1	12.5	37.5
Hermes	11.8	35.3	5.9	35.3	17.6	64.7	11.8	35.3
Selective	12.0	46.0	5.9	35.3	17.6	70.6	11.8	49.0
LRU (LLM)	12.5	37.5	5.9	35.3	17.6	64.7	11.8	41.2
FIFO	11.8	52.9	5.9	41.2	17.6	64.7	11.8	47.1
LRU (rule)	11.8	47.1	5.9	29.4	17.6	82.4	11.8	58.8

violation@3（%）；模型端点见主成绩协议与原表 17。

ARCBench original results · Table 22

S4 压缩预算扫描

请按该表配置独立阅读，与主成绩 S1 分开比较。

S4 压缩预算扫描
分组	配置	题例数	完整历史 %	已有违规 %	仅压缩后 %	压缩后总计 %	规则完整 %	规则部分 %	规则缺失 %	仅完整历史 %
All	2,500	32	15.6 (5)	12.5 (4)	15.6 (5)	28.1 (9)	100.0	0.0	0.0	3.1 (1)
All	1,200	34	14.7 (5)	11.8 (4)	14.7 (5)	26.5 (9)	97.1	2.9	0.0	2.9 (1)
All	750	33	15.2 (5)	9.1 (3)	15.2 (5)	24.2 (8)	98.0	1.0	1.0	6.1 (2)
All	300	34	14.7 (5)	11.8 (4)	35.3 (12)	47.1 (16)	2.9	2.0	95.1	2.9 (1)
All	200	34	14.7 (5)	8.8 (3)	41.2 (14)	50.0 (17)	0.0	0.0	100.0	5.9 (2)
DeepSeek	2,500	17	11.8 (2)	11.8 (2)	11.8 (2)	23.5 (4)	100.0	0.0	0.0	0.0 (0)
DeepSeek	1,200	17	11.8 (2)	11.8 (2)	11.8 (2)	23.5 (4)	98.0	2.0	0.0	0.0 (0)
DeepSeek	750	17	11.8 (2)	11.8 (2)	17.6 (3)	29.4 (5)	98.0	0.0	2.0	0.0 (0)
DeepSeek	300	17	11.8 (2)	5.9 (1)	35.3 (6)	41.2 (7)	5.9	3.9	90.2	5.9 (1)
DeepSeek	200	17	11.8 (2)	5.9 (1)	41.2 (7)	47.1 (8)	0.0	0.0	100.0	5.9 (1)
GLM	2,500	15	20.0 (3)	13.3 (2)	20.0 (3)	33.3 (5)	100.0	0.0	0.0	6.7 (1)
GLM	1,200	17	17.6 (3)	11.8 (2)	17.6 (3)	29.4 (5)	96.1	3.9	0.0	5.9 (1)
GLM	750	16	18.8 (3)	6.3 (1)	12.5 (2)	18.8 (3)	97.9	2.1	0.0	12.5 (2)
GLM	300	17	17.6 (3)	17.6 (3)	35.3 (6)	52.9 (9)	0.0	0.0	100.0	0.0 (0)
GLM	200	17	17.6 (3)	11.8 (2)	41.2 (7)	52.9 (9)	0.0	0.0	100.0	5.9 (1)

Hermes、c1/c3、容量注入；仅 DeepSeek 与 GLM。预算以估计 token 数表示。

ARCBench original results · Table 23

S2–S4 逐运行率与规则状态

请按该表配置独立阅读，与主成绩 S1 分开比较。

S2–S4 逐运行率与规则状态
分组	条件	完整违规 %	压缩违规 %	完整保留占比 %	部分保留占比 %	缺失占比 %	完整保留时违规 %	部分保留时违规 %	缺失时违规 %
S2: delivery channel	c1	19.8	38.5	74.0	24.0	2.1	36.6	39.1	100.0
S2: delivery channel	c2	15.1	28.0	69.9	25.8	4.3	20.0	37.5	100.0
S2: delivery channel	c3	0.0	21.3	65.7	29.6	4.6	16.9	25.0	60.0
S2: delivery channel	c4	0.0	14.8	71.3	27.8	0.9	11.7	20.0	100.0
S3: compaction strategy	Generative	9.5	14.4	91.2	7.8	0.9	12.6	27.3	76.9
S3: compaction strategy	Pi + RP	9.5	6.5	99.0	1.0	0.0	6.5	0.0	–
S3: compaction strategy	Pi	9.5	9.0	96.0	4.0	0.0	9.3	0.0	–
S3: compaction strategy	Structured	9.5	11.4	94.5	5.0	0.5	11.6	10.0	0.0
S3: compaction strategy	TRACE	9.7	11.8	93.3	5.1	1.5	9.9	20.0	100.0
S3: compaction strategy	ACON-UTCO	9.3	13.2	90.7	8.8	0.5	10.8	33.3	100.0
S3: compaction strategy	ACON-UT	9.6	19.2	96.0	3.5	0.5	17.4	57.1	100.0
S3: compaction strategy	Hermes	9.3	29.4	69.6	27.0	3.4	26.8	30.9	71.4
S3: compaction strategy	Selective	9.4	38.4	15.9	0.0	84.1	20.6	–	41.8
S3: compaction strategy	LRU (LLM)	9.5	27.9	36.3	0.0	63.7	19.2	–	32.8
S3: compaction strategy	FIFO	9.3	47.1	0.0	0.0	100.0	–	–	47.1
S3: compaction strategy	LRU (rule)	9.3	40.2	11.8	0.0	88.2	25.0	–	42.2
S4: compaction budget constraint (tokens)	2,500	9.4	18.8	100.0	0.0	0.0	18.8	–	–
S4: compaction budget constraint (tokens)	1,200	8.8	18.6	97.1	2.9	0.0	17.2	66.7	–
S4: compaction budget constraint (tokens)	750	9.1	14.1	98.0	1.0	1.0	14.4	0.0	0.0
S4: compaction budget constraint (tokens)	300	8.8	37.3	2.9	2.0	95.1	0.0	0.0	39.2
S4: compaction budget constraint (tokens)	200	8.8	41.2	0.0	0.0	100.0	–	–	41.2

逐运行比例；破折号仍表示缺失或不适用，不能当作 0。

ARCBench original results · Table 24

S2–S4 行为解释

请按该表配置独立阅读，与主成绩 S1 分开比较。

S2–S4 行为解释
分组	条件	当作批准 %	范围解释 %	未提规则 %	承认仍违规 %
S2: delivery channel	c1	72.7	9.1	18.2	0.0
S2: delivery channel	c2	71.4	0.0	0.0	28.6
S2: delivery channel	c3	18.2	0.0	54.5	27.3
S2: delivery channel	c4	25.0	12.5	25.0	37.5
S3: compaction strategy	Generative	62.3	4.9	14.8	18.0
S3: compaction strategy	Pi + RP	50.0	0.0	0.0	50.0
S3: compaction strategy	Pi	50.0	0.0	0.0	50.0
S3: compaction strategy	Structured	71.4	0.0	0.0	28.6
S3: compaction strategy	TRACE	50.0	0.0	0.0	50.0
S3: compaction strategy	ACON-UTCO	87.5	0.0	0.0	12.5
S3: compaction strategy	ACON-UT	75.0	12.5	6.3	6.3
S3: compaction strategy	Hermes	45.5	4.5	36.4	13.6
S3: compaction strategy	Selective	100.0	0.0	0.0	0.0
S3: compaction strategy	LRU (LLM)	100.0	0.0	0.0	0.0
S3: compaction strategy	FIFO	–	–	–	–
S3: compaction strategy	LRU (rule)	100.0	0.0	0.0	0.0
S4: compaction budget constraint (tokens)	2,500	90.0	10.0	0.0	0.0
S4: compaction budget constraint (tokens)	1,200	88.9	0.0	0.0	11.1
S4: compaction budget constraint (tokens)	750	100.0	0.0	0.0	0.0
S4: compaction budget constraint (tokens)	300	–	–	–	–
S4: compaction budget constraint (tokens)	200	–	–	–	–

仅统计推理可读、保留规则且只在压缩后违规的运行。

ARCBench original results · Table 25

S5 添加内容实验

请按该表配置独立阅读，与主成绩 S1 分开比较。

S5 添加内容实验
分组	配置	题例数	完整历史 %	已有违规 %	仅压缩后 %	压缩后总计 %	规则完整 %	规则部分 %	规则缺失 %	仅完整历史 %
All	None	135	9.6 (13)	8.9 (12)	28.9 (39)	37.8 (51)	70.1	26.9	3.0	0.7 (1)
All	V	134	9.7 (13)	9.0 (12)	25.4 (34)	34.3 (46)	71.9	22.9	5.2	0.7 (1)
All	I	134	9.7 (13)	9.0 (12)	37.3 (50)	46.3 (62)	42.5	26.4	31.1	0.7 (1)
All	V+I	134	9.7 (13)	9.7 (13)	31.3 (42)	41.0 (55)	54.5	23.4	22.1	0.0 (0)
DeepSeek	None	34	8.8 (3)	8.8 (3)	17.6 (6)	26.5 (9)	98.0	2.0	0.0	0.0 (0)
DeepSeek	V	34	8.8 (3)	8.8 (3)	14.7 (5)	23.5 (8)	98.0	1.0	1.0	0.0 (0)
DeepSeek	I	34	8.8 (3)	8.8 (3)	23.5 (8)	32.4 (11)	54.9	16.7	28.4	0.0 (0)
DeepSeek	V+I	34	8.8 (3)	8.8 (3)	23.5 (8)	32.4 (11)	71.6	11.8	16.7	0.0 (0)
GPT-Luna	None	33	6.1 (2)	6.1 (2)	27.3 (9)	33.3 (11)	48.5	45.5	6.1	0.0 (0)
GPT-Luna	V	33	6.1 (2)	6.1 (2)	24.2 (8)	30.3 (10)	48.5	42.4	9.1	0.0 (0)
GPT-Luna	I	33	6.1 (2)	6.1 (2)	39.4 (13)	45.5 (15)	2.0	16.2	81.8	0.0 (0)
GPT-Luna	V+I	33	6.1 (2)	6.1 (2)	27.3 (9)	33.3 (11)	16.2	20.2	63.6	0.0 (0)
Qwen	None	34	14.7 (5)	11.8 (4)	50.0 (17)	61.8 (21)	35.3	58.8	5.9	2.9 (1)
Qwen	V	34	14.7 (5)	11.8 (4)	52.9 (18)	64.7 (22)	43.1	46.1	10.8	2.9 (1)
Qwen	I	34	14.7 (5)	11.8 (4)	61.8 (21)	73.5 (25)	22.5	64.7	12.7	2.9 (1)
Qwen	V+I	34	14.7 (5)	14.7 (5)	55.9 (19)	70.6 (24)	35.3	56.9	7.8	0.0 (0)
GLM	None	34	8.8 (3)	8.8 (3)	20.6 (7)	29.4 (10)	98.0	2.0	0.0	0.0 (0)
GLM	V	33	9.1 (3)	9.1 (3)	9.1 (3)	18.2 (6)	98.0	2.0	0.0	0.0 (0)
GLM	I	33	9.1 (3)	9.1 (3)	24.2 (8)	33.3 (11)	90.9	7.1	2.0	0.0 (0)
GLM	V+I	33	9.1 (3)	9.1 (3)	18.2 (6)	27.3 (9)	94.9	4.0	1.0	0.0 (0)

violation@3；V 为容量注入，I 为 legacy 压缩器目标注入。新增内容使用成对 S1 完整历史基线。

ARCBench original results · Table 26

S6 攻击构造

请按该表配置独立阅读，与主成绩 S1 分开比较。

S6 攻击构造
攻击	题例数	完整 violation@6 %	压缩 violation@6 %	压缩逐运行 %
None	17	11.8	23.5	15.7
Legacy	17	17.6	35.3	17.6
Naive	17	23.5	41.2	19.6
Escape	17	11.8	58.8	30.4
Ignore	17	17.6	29.4	19.6
Fake completion	17	23.5	41.2	16.7
Combined	17	23.5	29.4	19.6

仅 GLM、17 个题例、六次重复、无防御。这里为 violation@6，不是主成绩 violation@3。

ARCBench original results · Table 27

S7 有攻击与无攻击时的防御

请按该表配置独立阅读，与主成绩 S1 分开比较。

S7 有攻击与无攻击时的防御
分组	配置	题例数	完整历史 %	已有违规 %	仅压缩后 %	压缩后总计 %	规则完整 %	规则部分 %	规则缺失 %
No attack	None	17	11.8 (2)	11.8 (2)	11.8 (2)	23.5 (4)	96.1	3.9	0.0
No attack	Isolation	17	11.8 (2)	11.8 (2)	11.8 (2)	23.5 (4)	94.1	5.9	0.0
No attack	Instructional	17	11.8 (2)	11.8 (2)	17.6 (3)	29.4 (5)	97.1	2.9	0.0
No attack	Sandwich	17	11.8 (2)	11.8 (2)	17.6 (3)	29.4 (5)	93.1	6.9	0.0
No attack	Rule pres.	17	11.8 (2)	11.8 (2)	11.8 (2)	23.5 (4)	100.0	0.0	0.0
Escape	None	17	11.8 (2)	11.8 (2)	47.1 (8)	58.8 (10)	66.7	13.7	19.6
Escape	Isolation	17	11.8 (2)	11.8 (2)	29.4 (5)	41.2 (7)	85.3	6.9	7.8
Escape	Instructional	17	11.8 (2)	11.8 (2)	29.4 (5)	41.2 (7)	87.3	7.8	4.9
Escape	Sandwich	17	11.8 (2)	11.8 (2)	41.2 (7)	52.9 (9)	82.4	11.8	5.9
Escape	Rule pres.	17	11.8 (2)	11.8 (2)	17.6 (3)	29.4 (5)	99.0	1.0	0.0

GLM、17 个题例、六次重复、c1/c3、Hermes；Full/Own/After/Total 使用 violation@6。

ARCBench original results · Table 28

策略与添加内容的组合

请按该表配置独立阅读，与主成绩 S1 分开比较。

策略与添加内容的组合
分组	配置	完整历史 %	已有违规 %	仅压缩后 %	压缩后总计 %	规则完整 %	规则部分 %	规则缺失 %
Added content: None	Generative	12.0	10.0	11.8	21.8	91.2	7.8	0.9
Added content: None	Pi + RP	11.9	6.0	3.0	9.0	99.0	1.0	0.0
Added content: None	Pi	11.9	9.0	6.0	14.9	96.0	4.0	0.0
Added content: None	Structured	11.9	9.0	6.0	14.9	94.5	5.0	0.5
Added content: None	TRACE	12.3	10.8	7.7	18.5	93.3	5.1	1.5
Added content: None	ACON-UTCO	11.8	11.8	8.8	20.6	90.7	8.8	0.5
Added content: None	ACON-UT	12.1	12.1	19.7	31.8	96.0	3.5	0.5
Added content: None	Hermes	11.8	11.8	30.9	42.6	69.6	27.0	3.4
Added content: None	Selective	11.8	9.9	40.4	50.2	15.9	0.0	84.1
Added content: None	LRU (LLM)	11.9	9.0	35.8	44.8	36.3	0.0	63.7
Added content: None	FIFO	11.8	10.3	41.2	51.5	0.0	0.0	100.0
Added content: None	LRU (rule)	11.8	10.3	44.1	54.4	11.8	0.0	88.2
Added content: V	Generative	14.4	9.9	14.4	24.4	90.9	6.7	2.4
Added content: V	Pi + RP	14.9	6.0	3.0	9.0	100.0	0.0	0.0
Added content: V	Pi	15.2	10.6	12.1	22.7	97.5	2.5	0.0
Added content: V	Structured	14.9	6.0	11.9	17.9	93.5	5.0	1.5
Added content: V	TRACE	14.8	9.8	14.8	24.6	91.8	3.3	4.9
Added content: V	ACON-UTCO	14.7	11.8	17.6	29.4	92.6	5.9	1.5
Added content: V	ACON-UT	14.7	14.7	10.3	25.0	92.2	5.9	2.0
Added content: V	Hermes	11.9	10.4	31.3	41.8	68.7	23.9	7.5
Added content: V	Selective	15.1	10.1	34.7	44.7	17.9	0.0	82.1
Added content: V	LRU (LLM)	15.9	11.1	27.0	38.1	37.6	0.0	62.4
Added content: V	FIFO	14.7	7.4	44.1	51.5	0.0	0.0	100.0
Added content: V	LRU (rule)	14.7	11.8	32.4	44.1	17.6	0.0	82.4
Added content: I	Generative	13.5	9.0	27.1	36.1	69.7	13.9	16.3
Added content: I	Pi + RP	13.4	10.4	7.5	17.9	99.5	0.5	0.0
Added content: I	Pi	13.4	7.5	23.9	31.3	73.6	9.0	17.4
Added content: I	Structured	13.6	7.6	24.2	31.8	76.3	8.1	15.7
Added content: I	TRACE	14.1	7.8	23.4	31.3	70.3	14.6	15.1
Added content: I	ACON-UTCO	13.4	9.0	34.3	43.3	60.7	19.9	19.4
Added content: I	ACON-UT	13.6	10.6	34.8	45.5	64.1	20.7	15.2
Added content: I	Hermes	13.2	10.3	41.2	51.5	44.1	24.5	31.4
Added content: I	Selective	13.6	12.1	35.7	47.7	11.2	0.0	88.8
Added content: I	LRU (LLM)	14.3	12.7	28.6	41.3	22.8	0.0	77.2
Added content: I	FIFO	13.2	11.8	44.1	55.9	0.0	0.0	100.0
Added content: I	LRU (rule)	13.2	11.8	33.8	45.6	11.8	0.0	88.2
Added content: V+I	Generative	9.5	7.6	25.4	33.0	72.9	12.4	14.8
Added content: V+I	Pi + RP	9.1	7.6	6.1	13.6	100.0	0.0	0.0
Added content: V+I	Pi	9.0	6.0	17.9	23.9	83.6	8.5	8.0
Added content: V+I	Structured	9.1	6.1	22.7	28.8	82.3	7.1	10.6
Added content: V+I	TRACE	9.5	7.9	27.0	34.9	65.1	14.8	20.1
Added content: V+I	ACON-UTCO	9.4	7.8	32.8	40.6	64.1	17.2	18.8
Added content: V+I	ACON-UT	9.0	6.0	34.3	40.3	63.2	17.4	19.4
Added content: V+I	Hermes	11.8	11.8	36.8	48.5	52.0	21.6	26.5
Added content: V+I	Selective	9.1	8.6	39.9	48.5	15.8	0.0	84.2
Added content: V+I	LRU (LLM)	9.7	9.7	33.9	43.5	31.2	0.0	68.8
Added content: V+I	FIFO	8.8	7.4	47.1	54.4	0.0	0.0	100.0
Added content: V+I	LRU (rule)	8.8	8.8	38.2	47.1	17.6	0.0	82.4

violation@3；V=容量注入，I=压缩器目标注入；类别汇总与单独策略分行保留。

ARCBench original results · Table 29

S5/S7 与策略逐运行率

请按该表配置独立阅读，与主成绩 S1 分开比较。

S5/S7 与策略逐运行率
分组	条件	完整违规 %	压缩违规 %	完整保留占比 %	部分保留占比 %	缺失占比 %	完整保留时违规 %	部分保留时违规 %	缺失时违规 %
S5: added content (Hermes, four models)	None	8.1	25.2	70.1	26.9	3.0	21.1	29.4	83.3
S5: added content (Hermes, four models)	V	8.2	23.1	71.9	22.9	5.2	16.3	37.0	57.1
S5: added content (Hermes, four models)	I	8.2	35.6	42.5	26.4	31.1	22.2	49.1	42.4
S5: added content (Hermes, four models)	V+I	8.2	27.9	54.5	23.4	22.1	17.4	41.5	39.3
S7: no attack, by defense (GLM)	None	6.9	15.7	96.1	3.9	0.0	13.3	75.0	–
S7: no attack, by defense (GLM)	Isolation	6.9	18.6	94.1	5.9	0.0	17.7	33.3	–
S7: no attack, by defense (GLM)	Instructional	6.9	14.7	97.1	2.9	0.0	13.1	66.7	–
S7: no attack, by defense (GLM)	Sandwich	6.9	15.7	93.1	6.9	0.0	13.7	42.9	–
S7: no attack, by defense (GLM)	Rule pres.	6.9	14.7	100.0	0.0	0.0	14.7	–	–
S7: escape, by defense (GLM)	None	6.9	30.4	66.7	13.7	19.6	26.5	28.6	45.0
S7: escape, by defense (GLM)	Isolation	6.9	21.6	85.3	6.9	7.8	23.0	0.0	25.0
S7: escape, by defense (GLM)	Instructional	6.9	21.6	87.3	7.8	4.9	20.2	0.0	80.0
S7: escape, by defense (GLM)	Sandwich	6.9	20.6	82.4	11.8	5.9	15.5	33.3	66.7
S7: escape, by defense (GLM)	Rule pres.	6.9	16.7	99.0	1.0	0.0	15.8	100.0	–
Strategy study: family by added content	Generative None	9.5	14.4	91.2	7.8	0.9	12.6	27.3	76.9
Strategy study: family by added content	Generative V	9.1	15.0	90.9	6.7	2.4	12.3	32.3	70.6
Strategy study: family by added content	Generative I	9.5	24.6	69.7	13.9	16.3	14.4	45.4	50.4
Strategy study: family by added content	Generative V+I	7.4	22.8	72.9	12.4	14.8	15.1	40.4	46.6
Strategy study: family by added content	Selective None	9.4	38.4	15.9	0.0	84.1	20.6	–	41.8
Strategy study: family by added content	Selective V	9.0	36.7	17.9	0.0	82.1	18.7	–	40.6
Strategy study: family by added content	Selective I	9.5	40.2	11.2	0.0	88.8	13.4	–	43.6
Strategy study: family by added content	Selective V+I	7.1	38.9	15.8	0.0	84.2	17.0	–	43.0

逐运行率与规则状态占比；S7 仅 GLM。不是 violation@3 或 violation@6。

ARCBench original results · Table 30

S5–S7 行为解释

请按该表配置独立阅读，与主成绩 S1 分开比较。

S5–S7 行为解释
分组	条件	当作批准 %	范围解释 %	未提规则 %	承认仍违规 %
S5: added content	None	45.9	5.4	27.0	21.6
S5: added content	V	40.0	6.7	26.7	26.7
S5: added content	I	60.7	7.1	17.9	14.3
S5: added content	V+I	42.3	11.5	38.5	7.7
S6: attack, no defense	None	100.0	0.0	0.0	0.0
S6: attack, no defense	Escape	61.5	15.4	15.4	7.7
S7: defense under escape	Isolation	90.9	0.0	9.1	0.0
S7: defense under escape	Instructional	83.3	0.0	16.7	0.0
S7: defense under escape	Sandwich	50.0	12.5	25.0	12.5
S7: defense under escape	Rule pres.	71.4	14.3	0.0	14.3

推理可读、保留规则且只在压缩后违规的运行。

ARCBench original results · Table 31

原始发布

2026 年 11 月 2 日原论文 v1，11 月 4 日核查。主成绩为 S1 基线，其余模型、渠道、策略、预算与攻防实验独立标注。

核对于 2026-11-04

已收录的评测资料

这条来源记录收录了 8 条数值成绩。来源区分了 4 个模型身份或配置。同一模型可能有多条记录；比较前请核对具体版本、工具和运行预算。查看成绩及逐条证据 →

报告指标: S1 violation@3 (at least one violation across three valid repetitions)
原始单位: percent
指标方向: 数值越低越好

从这些资料开始

收录来源 · arxiv.org

引用文档及日期（1）

ARCBench · original v1 evaluation · 2026-11-02

评测信息与证据

来源: Model reports
发布者: 未收录
评测发布日期: 2026-11-02
采集日期: 未收录
模态: 未收录

本站保留来源报告的结果，不代表独立复测。请先核对任务版本、评测协议、分数单位和报告日期；不同运行条件不能直接合并为统一排名。

模型	数值	单位	协议	报告日期与口径	来源
GPT-Luna	6.1	percent	S1; Full history, no compaction; no injection or defense; personal cases c1/c2, organizational cases c3/c4; 33 valid task-channel settings; violation@3; same endpoint agent and compactor: ~openai/gpt-luna-latest; pi revision 97fa14e3; temperature 0.7, seed 7, four tool steps; source publication date, exact run dates unknown	2026-11-02 · document_publication	ARCBench · original v1 evaluation
DeepSeek-V4-Flash-0731	8.8	percent	S1; Full history, no compaction; no injection or defense; personal cases c1/c2, organizational cases c3/c4; 34 valid task-channel settings; violation@3; same endpoint agent and compactor: deepseek/deepseek-v4-flash-0731; pi revision 97fa14e3; temperature 0.7, seed 7, four tool steps; source publication date, exact run dates unknown	2026-11-02 · document_publication	ARCBench · original v1 evaluation
Qwen3.7-Flash	14.7	percent	S1; Full history, no compaction; no injection or defense; personal cases c1/c2, organizational cases c3/c4; 34 valid task-channel settings; violation@3; same endpoint agent and compactor: qwen/qwen3.7-flash; pi revision 97fa14e3; temperature 0.7, seed 7, four tool steps; source publication date, exact run dates unknown	2026-11-02 · document_publication	ARCBench · original v1 evaluation
GLM-5.3-Flash	8.8	percent	S1; Full history, no compaction; no injection or defense; personal cases c1/c2, organizational cases c3/c4; 34 valid task-channel settings; violation@3; same endpoint agent and compactor: z-ai/glm-5.3-flash; pi revision 97fa14e3; temperature 0.7, seed 7, four tool steps; source publication date, exact run dates unknown	2026-11-02 · document_publication	ARCBench · original v1 evaluation
GPT-Luna	33.3	percent	S1; Hermes compaction, no extra budget; no injection or defense; personal cases c1/c2, organizational cases c3/c4; 33 valid task-channel settings; violation@3; same endpoint agent and compactor: ~openai/gpt-luna-latest; pi revision 97fa14e3; temperature 0.7, seed 7, four tool steps; source publication date, exact run dates unknown	2026-11-02 · document_publication	ARCBench · original v1 evaluation
DeepSeek-V4-Flash-0731	26.5	percent	S1; Hermes compaction, no extra budget; no injection or defense; personal cases c1/c2, organizational cases c3/c4; 34 valid task-channel settings; violation@3; same endpoint agent and compactor: deepseek/deepseek-v4-flash-0731; pi revision 97fa14e3; temperature 0.7, seed 7, four tool steps; source publication date, exact run dates unknown	2026-11-02 · document_publication	ARCBench · original v1 evaluation
Qwen3.7-Flash	61.8	percent	S1; Hermes compaction, no extra budget; no injection or defense; personal cases c1/c2, organizational cases c3/c4; 34 valid task-channel settings; violation@3; same endpoint agent and compactor: qwen/qwen3.7-flash; pi revision 97fa14e3; temperature 0.7, seed 7, four tool steps; source publication date, exact run dates unknown	2026-11-02 · document_publication	ARCBench · original v1 evaluation
GLM-5.3-Flash	29.4	percent	S1; Hermes compaction, no extra budget; no injection or defense; personal cases c1/c2, organizational cases c3/c4; 34 valid task-channel settings; violation@3; same endpoint agent and compactor: z-ai/glm-5.3-flash; pi revision 97fa14e3; temperature 0.7, seed 7, four tool steps; source publication date, exact run dates unknown	2026-11-02 · document_publication	ARCBench · original v1 evaluation

压缩历史后的规则

越低越好；题例不等于运行

保留模型端点与框架设置

保留规则是另一项测量

S1：完整历史与 Hermes

S1 模型与规则渠道

S1 逐运行结果

S1 行为归因数量

S3 压缩策略比较

S3 各模型的策略结果

S4 压缩预算扫描

S2–S4 逐运行率与规则状态

S2–S4 行为解释

S5 添加内容实验

S6 攻击构造

S7 有攻击与无攻击时的防御

策略与添加内容的组合

S5/S7 与策略逐运行率

S5–S7 行为解释

原始发布

已收录的评测资料

从这些资料开始

评测信息与证据

报告成绩 8

相关评测探索

继续阅读

继续探索