ARCBench — Model reports · AI Benchmark Radar

AI Benchmark Radar / Equality Machine · Reviewed 2026-11-04

Original paper v1, 2 November 2026; reviewed 4 November. Primary S1 baseline, with model/channel, strategy, budget and attack-defense details separated.

Rules after a shortened history

Seventeen fixtures cover data disclosure, restricted access, unauthorized actions and required controls. Recorded tool calls determine violations inside the pi agent harness.

ARCBench · original paper v1

Lower is better; cases are not runs

Violation@3 counts settings with any violation over three valid repetitions. S1 retains 135 settings; six invalid GPT-Luna runs are excluded. Full and compacted histories are paired.

ARCBench · original paper v1

Keep the exact endpoint and harness

Agent and compactor share the endpoint. S1 uses Hermes, no attack or defense, temperature 0.7 and four tool steps. Actual run dates are unknown.

ARCBench · original paper v1

Retention is a different measurement

Rule-state and behavior labels come from separate judges. Mocked tools and short fixtures constrain generalization. Violation@6 and run-level rates cannot be compared as violation@3.

ARCBench · original paper v1

S1: full-history versus Hermes

Case-level violations and run-level rule states have different denominators.

S1 published baseline
Setting	Full %	Own %	After only %	Compacted total %	Retained %	Partial %	Absent %
All	9.6	8.9	28.9	37.8	70.1	26.9	3.0
DeepSeek	8.8	8.8	17.6	26.5	98.0	2.0	0.0
GPT-Luna	6.1	6.1	27.3	33.3	48.5	45.5	6.1
Qwen	14.7	11.8	50.0	61.8	35.3	58.8	5.9
GLM	8.8	8.8	20.6	29.4	98.0	2.0	0.0

Full/Own/After/Total are violation@3 case percentages; the last three columns classify compacted runs.

ARCBench original results · Table 3

S1 by model and delivery channel

Read this configuration independently from the primary S1 comparison.

S1 by model and delivery channel
Group	Setting	Cases	Full %	Own %	After only %	Compacted total %	Retained %	Partial %	Absent %	Only full %
All	All	135	9.6 (13)	8.9 (12)	28.9 (39)	37.8 (51)	70.1	26.9	3.0	0.7 (1)
All	c1	32	25.0 (8)	25.0 (8)	31.3 (10)	56.3 (18)	74.0	24.0	2.1	0.0 (0)
All	c2	31	16.1 (5)	12.9 (4)	32.3 (10)	45.2 (14)	69.9	25.8	4.3	3.2 (1)
All	c3	36	0.0 (0)	0.0 (0)	30.6 (11)	30.6 (11)	65.7	29.6	4.6	0.0 (0)
All	c4	36	0.0 (0)	0.0 (0)	22.2 (8)	22.2 (8)	71.3	27.8	0.9	0.0 (0)
DeepSeek	DeepSeek	34	8.8 (3)	8.8 (3)	17.6 (6)	26.5 (9)	98.0	2.0	0.0	0.0 (0)
DeepSeek	c1	8	25.0 (2)	25.0 (2)	37.5 (3)	62.5 (5)	95.8	4.2	0.0	0.0 (0)
DeepSeek	c2	8	12.5 (1)	12.5 (1)	25.0 (2)	37.5 (3)	100.0	0.0	0.0	0.0 (0)
DeepSeek	c3	9	0.0 (0)	0.0 (0)	11.1 (1)	11.1 (1)	96.3	3.7	0.0	0.0 (0)
DeepSeek	c4	9	0.0 (0)	0.0 (0)	0.0 (0)	0.0 (0)	100.0	0.0	0.0	0.0 (0)
GPT-Luna	GPT-Luna	33	6.1 (2)	6.1 (2)	27.3 (9)	33.3 (11)	48.5	45.5	6.1	0.0 (0)
GPT-Luna	c1	8	12.5 (1)	12.5 (1)	12.5 (1)	25.0 (2)	70.8	29.2	0.0	0.0 (0)
GPT-Luna	c2	7	14.3 (1)	14.3 (1)	14.3 (1)	28.6 (2)	47.6	47.6	4.8	0.0 (0)
GPT-Luna	c3	9	0.0 (0)	0.0 (0)	44.4 (4)	44.4 (4)	40.7	40.7	18.5	0.0 (0)
GPT-Luna	c4	9	0.0 (0)	0.0 (0)	33.3 (3)	33.3 (3)	37.0	63.0	0.0	0.0 (0)
Qwen	Qwen	34	14.7 (5)	11.8 (4)	50.0 (17)	61.8 (21)	35.3	58.8	5.9	2.9 (1)
Qwen	c1	8	37.5 (3)	37.5 (3)	37.5 (3)	75.0 (6)	29.2	62.5	8.3	0.0 (0)
Qwen	c2	8	25.0 (2)	12.5 (1)	50.0 (4)	62.5 (5)	29.2	58.3	12.5	12.5 (1)
Qwen	c3	9	0.0 (0)	0.0 (0)	55.6 (5)	55.6 (5)	29.6	70.4	0.0	0.0 (0)
Qwen	c4	9	0.0 (0)	0.0 (0)	55.6 (5)	55.6 (5)	51.9	44.4	3.7	0.0 (0)
GLM	GLM	34	8.8 (3)	8.8 (3)	20.6 (7)	29.4 (10)	98.0	2.0	0.0	0.0 (0)
GLM	c1	8	25.0 (2)	25.0 (2)	37.5 (3)	62.5 (5)	100.0	0.0	0.0	0.0 (0)
GLM	c2	8	12.5 (1)	12.5 (1)	37.5 (3)	50.0 (4)	100.0	0.0	0.0	0.0 (0)
GLM	c3	9	0.0 (0)	0.0 (0)	11.1 (1)	11.1 (1)	96.3	3.7	0.0	0.0 (0)
GLM	c4	9	0.0 (0)	0.0 (0)	0.0 (0)	0.0 (0)	96.3	3.7	0.0	0.0 (0)

Parentheses contain case counts. c1/c2 are personal channels; c3/c4 organizational. GPT-Luna has 33 valid settings.

ARCBench original results · Table 18

S1 run-level outcomes

Read this configuration independently from the primary S1 comparison.

S1 run-level outcomes
Group	Run condition	Share %	No violation %	Violation %
All	Full context	–	91.9	8.1
All	Compacted, rule retained	70.1	78.9	21.1
All	Compacted, partially retained	26.9	70.6	29.4
All	Compacted, rule absent	3.0	16.7	83.3
DeepSeek	Full context	–	93.1	6.9
DeepSeek	Compacted, rule retained	98.0	83.0	17.0
DeepSeek	Compacted, partially retained	2.0	100.0	0.0
DeepSeek	Compacted, rule absent	0.0	–	–
GPT-Luna	Full context	–	96.0	4.0
GPT-Luna	Compacted, rule retained	48.5	79.2	20.8
GPT-Luna	Compacted, partially retained	45.5	75.6	24.4
GPT-Luna	Compacted, rule absent	6.1	33.3	66.7
Qwen	Full context	–	86.3	13.7
Qwen	Compacted, rule retained	35.3	52.8	47.2
Qwen	Compacted, partially retained	58.8	65.0	35.0
Qwen	Compacted, rule absent	5.9	0.0	100.0
GLM	Full context	–	92.2	7.8
GLM	Compacted, rule retained	98.0	84.0	16.0
GLM	Compacted, partially retained	2.0	100.0	0.0
GLM	Compacted, rule absent	0.0	–	–

These are per-run percentages, not violation@3.

ARCBench original results · Table 19

S1 behavior attribution counts

Read this configuration independently from the primary S1 comparison.

S1 behavior attribution counts
Group	Model	Runs n	Approval	Scope	Unregistered	Acknowledged	Unobservable
Full context, agent’s own	All	30	21	2	5	1	1
Full context, agent’s own	DeepSeek	7	7	0	0	0	0
Full context, agent’s own	GPT-Luna	4	2	0	0	1	1
Full context, agent’s own	Qwen	11	4	2	5	0	0
Full context, agent’s own	GLM	8	8	0	0	0	0
Rule retained, agent’s own	All	20	14	1	2	0	3
Rule retained, agent’s own	DeepSeek	7	7	0	0	0	0
Rule retained, agent’s own	GPT-Luna	2	0	0	0	0	2
Rule retained, agent’s own	Qwen	2	0	1	1	0	0
Rule retained, agent’s own	GLM	9	7	0	1	0	1
Rule retained, only after compaction	All	40	17	2	10	8	3
Rule retained, only after compaction	DeepSeek	10	7	0	0	3	0
Rule retained, only after compaction	GPT-Luna	8	0	0	5	0	3
Rule retained, only after compaction	Qwen	15	6	1	4	4	0
Rule retained, only after compaction	GLM	7	4	1	1	1	0

Counts, not model-success scores.

ARCBench original results · Table 20

S3 strategy comparison

Read this configuration independently from the primary S1 comparison.

S3 strategy comparison
Setting	Cases	Full %	Own %	After only %	Compacted total %	Retained %	Partial %	Absent %	Only full %
Generative	468	12.0 (56)	10.0 (47)	11.8 (55)	21.8 (102)	91.2	7.8	0.9	1.9 (9)
Pi + RP	67	11.9 (8)	6.0 (4)	3.0 (2)	9.0 (6)	99.0	1.0	0.0	6.0 (4)
Pi	67	11.9 (8)	9.0 (6)	6.0 (4)	14.9 (10)	96.0	4.0	0.0	3.0 (2)
Structured	67	11.9 (8)	9.0 (6)	6.0 (4)	14.9 (10)	94.5	5.0	0.5	3.0 (2)
TRACE	65	12.3 (8)	10.8 (7)	7.7 (5)	18.5 (12)	93.3	5.1	1.5	1.5 (1)
ACON-UTCO	68	11.8 (8)	11.8 (8)	8.8 (6)	20.6 (14)	90.7	8.8	0.5	0.0 (0)
ACON-UT	66	12.1 (8)	12.1 (8)	19.7 (13)	31.8 (21)	96.0	3.5	0.5	0.0 (0)
Hermes	68	11.8 (8)	11.8 (8)	30.9 (21)	42.6 (29)	69.6	27.0	3.4	0.0 (0)
Selective	203	11.8 (24)	9.9 (20)	40.4 (82)	50.2 (102)	15.9	0.0	84.1	2.0 (4)
LRU (LLM)	67	11.9 (8)	9.0 (6)	35.8 (24)	44.8 (30)	36.3	0.0	63.7	3.0 (2)
FIFO	68	11.8 (8)	10.3 (7)	41.2 (28)	51.5 (35)	0.0	0.0	100.0	1.5 (1)
LRU (rule)	68	11.8 (8)	10.3 (7)	44.1 (30)	54.4 (37)	11.8	0.0	88.2	1.5 (1)

c1/c3, no attack and no extra budget; seven generative and three selective configurations. Family rows pool distinct conditions.

ARCBench original results · Table 21

S3 strategies by model

Read this configuration independently from the primary S1 comparison.

S3 strategies by model
Strategy	DeepSeek full	DeepSeek comp.	GPT-Luna full	GPT-Luna comp.	Qwen full	Qwen comp.	GLM full	GLM comp.
Generative	12.2	20.0	5.9	14.3	17.6	31.1	12.2	21.7
Pi + RP	11.8	11.8	5.9	5.9	17.6	5.9	12.5	12.5
Pi	11.8	11.8	5.9	11.8	17.6	17.6	12.5	18.8
Structured	11.8	11.8	5.9	5.9	17.6	29.4	12.5	12.5
TRACE	14.3	21.4	5.9	17.6	17.6	23.5	11.8	11.8
ACON-UTCO	11.8	23.5	5.9	5.9	17.6	29.4	11.8	23.5
ACON-UT	12.5	25.0	5.9	17.6	17.6	47.1	12.5	37.5
Hermes	11.8	35.3	5.9	35.3	17.6	64.7	11.8	35.3
Selective	12.0	46.0	5.9	35.3	17.6	70.6	11.8	49.0
LRU (LLM)	12.5	37.5	5.9	35.3	17.6	64.7	11.8	41.2
FIFO	11.8	52.9	5.9	41.2	17.6	64.7	11.8	47.1
LRU (rule)	11.8	47.1	5.9	29.4	17.6	82.4	11.8	58.8

violation@3 (%); model endpoints are preserved in primary protocols and the original Table 17.

ARCBench original results · Table 22

S4 budget sweep

Read this configuration independently from the primary S1 comparison.

S4 budget sweep
Group	Setting	Cases	Full %	Own %	After only %	Compacted total %	Retained %	Partial %	Absent %	Only full %
All	2,500	32	15.6 (5)	12.5 (4)	15.6 (5)	28.1 (9)	100.0	0.0	0.0	3.1 (1)
All	1,200	34	14.7 (5)	11.8 (4)	14.7 (5)	26.5 (9)	97.1	2.9	0.0	2.9 (1)
All	750	33	15.2 (5)	9.1 (3)	15.2 (5)	24.2 (8)	98.0	1.0	1.0	6.1 (2)
All	300	34	14.7 (5)	11.8 (4)	35.3 (12)	47.1 (16)	2.9	2.0	95.1	2.9 (1)
All	200	34	14.7 (5)	8.8 (3)	41.2 (14)	50.0 (17)	0.0	0.0	100.0	5.9 (2)
DeepSeek	2,500	17	11.8 (2)	11.8 (2)	11.8 (2)	23.5 (4)	100.0	0.0	0.0	0.0 (0)
DeepSeek	1,200	17	11.8 (2)	11.8 (2)	11.8 (2)	23.5 (4)	98.0	2.0	0.0	0.0 (0)
DeepSeek	750	17	11.8 (2)	11.8 (2)	17.6 (3)	29.4 (5)	98.0	0.0	2.0	0.0 (0)
DeepSeek	300	17	11.8 (2)	5.9 (1)	35.3 (6)	41.2 (7)	5.9	3.9	90.2	5.9 (1)
DeepSeek	200	17	11.8 (2)	5.9 (1)	41.2 (7)	47.1 (8)	0.0	0.0	100.0	5.9 (1)
GLM	2,500	15	20.0 (3)	13.3 (2)	20.0 (3)	33.3 (5)	100.0	0.0	0.0	6.7 (1)
GLM	1,200	17	17.6 (3)	11.8 (2)	17.6 (3)	29.4 (5)	96.1	3.9	0.0	5.9 (1)
GLM	750	16	18.8 (3)	6.3 (1)	12.5 (2)	18.8 (3)	97.9	2.1	0.0	12.5 (2)
GLM	300	17	17.6 (3)	17.6 (3)	35.3 (6)	52.9 (9)	0.0	0.0	100.0	0.0 (0)
GLM	200	17	17.6 (3)	11.8 (2)	41.2 (7)	52.9 (9)	0.0	0.0	100.0	5.9 (1)

Hermes, c1/c3, volume injection; DeepSeek and GLM only. Budget labels are estimated tokens.

ARCBench original results · Table 23

S2–S4 run-level rates and rule states

Read this configuration independently from the primary S1 comparison.

S2–S4 run-level rates and rule states
Group	Condition	Full violation %	Comp. violation %	Retained share %	Partial share %	Absent share %	Violation if retained %	Violation if partial %	Violation if absent %
S2: delivery channel	c1	19.8	38.5	74.0	24.0	2.1	36.6	39.1	100.0
S2: delivery channel	c2	15.1	28.0	69.9	25.8	4.3	20.0	37.5	100.0
S2: delivery channel	c3	0.0	21.3	65.7	29.6	4.6	16.9	25.0	60.0
S2: delivery channel	c4	0.0	14.8	71.3	27.8	0.9	11.7	20.0	100.0
S3: compaction strategy	Generative	9.5	14.4	91.2	7.8	0.9	12.6	27.3	76.9
S3: compaction strategy	Pi + RP	9.5	6.5	99.0	1.0	0.0	6.5	0.0	–
S3: compaction strategy	Pi	9.5	9.0	96.0	4.0	0.0	9.3	0.0	–
S3: compaction strategy	Structured	9.5	11.4	94.5	5.0	0.5	11.6	10.0	0.0
S3: compaction strategy	TRACE	9.7	11.8	93.3	5.1	1.5	9.9	20.0	100.0
S3: compaction strategy	ACON-UTCO	9.3	13.2	90.7	8.8	0.5	10.8	33.3	100.0
S3: compaction strategy	ACON-UT	9.6	19.2	96.0	3.5	0.5	17.4	57.1	100.0
S3: compaction strategy	Hermes	9.3	29.4	69.6	27.0	3.4	26.8	30.9	71.4
S3: compaction strategy	Selective	9.4	38.4	15.9	0.0	84.1	20.6	–	41.8
S3: compaction strategy	LRU (LLM)	9.5	27.9	36.3	0.0	63.7	19.2	–	32.8
S3: compaction strategy	FIFO	9.3	47.1	0.0	0.0	100.0	–	–	47.1
S3: compaction strategy	LRU (rule)	9.3	40.2	11.8	0.0	88.2	25.0	–	42.2
S4: compaction budget constraint (tokens)	2,500	9.4	18.8	100.0	0.0	0.0	18.8	–	–
S4: compaction budget constraint (tokens)	1,200	8.8	18.6	97.1	2.9	0.0	17.2	66.7	–
S4: compaction budget constraint (tokens)	750	9.1	14.1	98.0	1.0	1.0	14.4	0.0	0.0
S4: compaction budget constraint (tokens)	300	8.8	37.3	2.9	2.0	95.1	0.0	0.0	39.2
S4: compaction budget constraint (tokens)	200	8.8	41.2	0.0	0.0	100.0	–	–	41.2

Per-run rates; a dash remains missing or inapplicable, never zero.

ARCBench original results · Table 24

S2–S4 behavior labels

Read this configuration independently from the primary S1 comparison.

S2–S4 behavior labels
Group	Condition	Approval %	Scope %	Unregistered %	Acknowledged %
S2: delivery channel	c1	72.7	9.1	18.2	0.0
S2: delivery channel	c2	71.4	0.0	0.0	28.6
S2: delivery channel	c3	18.2	0.0	54.5	27.3
S2: delivery channel	c4	25.0	12.5	25.0	37.5
S3: compaction strategy	Generative	62.3	4.9	14.8	18.0
S3: compaction strategy	Pi + RP	50.0	0.0	0.0	50.0
S3: compaction strategy	Pi	50.0	0.0	0.0	50.0
S3: compaction strategy	Structured	71.4	0.0	0.0	28.6
S3: compaction strategy	TRACE	50.0	0.0	0.0	50.0
S3: compaction strategy	ACON-UTCO	87.5	0.0	0.0	12.5
S3: compaction strategy	ACON-UT	75.0	12.5	6.3	6.3
S3: compaction strategy	Hermes	45.5	4.5	36.4	13.6
S3: compaction strategy	Selective	100.0	0.0	0.0	0.0
S3: compaction strategy	LRU (LLM)	100.0	0.0	0.0	0.0
S3: compaction strategy	FIFO	–	–	–	–
S3: compaction strategy	LRU (rule)	100.0	0.0	0.0	0.0
S4: compaction budget constraint (tokens)	2,500	90.0	10.0	0.0	0.0
S4: compaction budget constraint (tokens)	1,200	88.9	0.0	0.0	11.1
S4: compaction budget constraint (tokens)	750	100.0	0.0	0.0	0.0
S4: compaction budget constraint (tokens)	300	–	–	–	–
S4: compaction budget constraint (tokens)	200	–	–	–	–

Among readable-reasoning violating runs that retain the rule and violate only after compaction.

ARCBench original results · Table 25

S5 added-content experiment

Read this configuration independently from the primary S1 comparison.

S5 added-content experiment
Group	Setting	Cases	Full %	Own %	After only %	Compacted total %	Retained %	Partial %	Absent %	Only full %
All	None	135	9.6 (13)	8.9 (12)	28.9 (39)	37.8 (51)	70.1	26.9	3.0	0.7 (1)
All	V	134	9.7 (13)	9.0 (12)	25.4 (34)	34.3 (46)	71.9	22.9	5.2	0.7 (1)
All	I	134	9.7 (13)	9.0 (12)	37.3 (50)	46.3 (62)	42.5	26.4	31.1	0.7 (1)
All	V+I	134	9.7 (13)	9.7 (13)	31.3 (42)	41.0 (55)	54.5	23.4	22.1	0.0 (0)
DeepSeek	None	34	8.8 (3)	8.8 (3)	17.6 (6)	26.5 (9)	98.0	2.0	0.0	0.0 (0)
DeepSeek	V	34	8.8 (3)	8.8 (3)	14.7 (5)	23.5 (8)	98.0	1.0	1.0	0.0 (0)
DeepSeek	I	34	8.8 (3)	8.8 (3)	23.5 (8)	32.4 (11)	54.9	16.7	28.4	0.0 (0)
DeepSeek	V+I	34	8.8 (3)	8.8 (3)	23.5 (8)	32.4 (11)	71.6	11.8	16.7	0.0 (0)
GPT-Luna	None	33	6.1 (2)	6.1 (2)	27.3 (9)	33.3 (11)	48.5	45.5	6.1	0.0 (0)
GPT-Luna	V	33	6.1 (2)	6.1 (2)	24.2 (8)	30.3 (10)	48.5	42.4	9.1	0.0 (0)
GPT-Luna	I	33	6.1 (2)	6.1 (2)	39.4 (13)	45.5 (15)	2.0	16.2	81.8	0.0 (0)
GPT-Luna	V+I	33	6.1 (2)	6.1 (2)	27.3 (9)	33.3 (11)	16.2	20.2	63.6	0.0 (0)
Qwen	None	34	14.7 (5)	11.8 (4)	50.0 (17)	61.8 (21)	35.3	58.8	5.9	2.9 (1)
Qwen	V	34	14.7 (5)	11.8 (4)	52.9 (18)	64.7 (22)	43.1	46.1	10.8	2.9 (1)
Qwen	I	34	14.7 (5)	11.8 (4)	61.8 (21)	73.5 (25)	22.5	64.7	12.7	2.9 (1)
Qwen	V+I	34	14.7 (5)	14.7 (5)	55.9 (19)	70.6 (24)	35.3	56.9	7.8	0.0 (0)
GLM	None	34	8.8 (3)	8.8 (3)	20.6 (7)	29.4 (10)	98.0	2.0	0.0	0.0 (0)
GLM	V	33	9.1 (3)	9.1 (3)	9.1 (3)	18.2 (6)	98.0	2.0	0.0	0.0 (0)
GLM	I	33	9.1 (3)	9.1 (3)	24.2 (8)	33.3 (11)	90.9	7.1	2.0	0.0 (0)
GLM	V+I	33	9.1 (3)	9.1 (3)	18.2 (6)	27.3 (9)	94.9	4.0	1.0	0.0 (0)

violation@3; V is volume and I is legacy compactor-targeted injection. Added-content rows reuse matched S1 full-history baselines.

ARCBench original results · Table 26

S6 attack constructions

Read this configuration independently from the primary S1 comparison.

S6 attack constructions
Attack	Cases	Full violation@6 %	Comp. violation@6 %	Comp. per-run %
None	17	11.8	23.5	15.7
Legacy	17	17.6	35.3	17.6
Naive	17	23.5	41.2	19.6
Escape	17	11.8	58.8	30.4
Ignore	17	17.6	29.4	19.6
Fake completion	17	23.5	41.2	16.7
Combined	17	23.5	29.4	19.6

GLM only; 17 cases, six repetitions; no defense. This is violation@6, not the primary violation@3.

ARCBench original results · Table 27

S7 defenses with and without attack

Read this configuration independently from the primary S1 comparison.

S7 defenses with and without attack
Group	Setting	Cases	Full %	Own %	After only %	Compacted total %	Retained %	Partial %	Absent %
No attack	None	17	11.8 (2)	11.8 (2)	11.8 (2)	23.5 (4)	96.1	3.9	0.0
No attack	Isolation	17	11.8 (2)	11.8 (2)	11.8 (2)	23.5 (4)	94.1	5.9	0.0
No attack	Instructional	17	11.8 (2)	11.8 (2)	17.6 (3)	29.4 (5)	97.1	2.9	0.0
No attack	Sandwich	17	11.8 (2)	11.8 (2)	17.6 (3)	29.4 (5)	93.1	6.9	0.0
No attack	Rule pres.	17	11.8 (2)	11.8 (2)	11.8 (2)	23.5 (4)	100.0	0.0	0.0
Escape	None	17	11.8 (2)	11.8 (2)	47.1 (8)	58.8 (10)	66.7	13.7	19.6
Escape	Isolation	17	11.8 (2)	11.8 (2)	29.4 (5)	41.2 (7)	85.3	6.9	7.8
Escape	Instructional	17	11.8 (2)	11.8 (2)	29.4 (5)	41.2 (7)	87.3	7.8	4.9
Escape	Sandwich	17	11.8 (2)	11.8 (2)	41.2 (7)	52.9 (9)	82.4	11.8	5.9
Escape	Rule pres.	17	11.8 (2)	11.8 (2)	17.6 (3)	29.4 (5)	99.0	1.0	0.0

GLM, 17 cases, six repetitions, c1/c3, Hermes. Full/Own/After/Total use violation@6.

ARCBench original results · Table 28

Strategy and added-content combinations

Read this configuration independently from the primary S1 comparison.

Strategy and added-content combinations
Group	Setting	Full %	Own %	After only %	Compacted total %	Retained %	Partial %	Absent %
Added content: None	Generative	12.0	10.0	11.8	21.8	91.2	7.8	0.9
Added content: None	Pi + RP	11.9	6.0	3.0	9.0	99.0	1.0	0.0
Added content: None	Pi	11.9	9.0	6.0	14.9	96.0	4.0	0.0
Added content: None	Structured	11.9	9.0	6.0	14.9	94.5	5.0	0.5
Added content: None	TRACE	12.3	10.8	7.7	18.5	93.3	5.1	1.5
Added content: None	ACON-UTCO	11.8	11.8	8.8	20.6	90.7	8.8	0.5
Added content: None	ACON-UT	12.1	12.1	19.7	31.8	96.0	3.5	0.5
Added content: None	Hermes	11.8	11.8	30.9	42.6	69.6	27.0	3.4
Added content: None	Selective	11.8	9.9	40.4	50.2	15.9	0.0	84.1
Added content: None	LRU (LLM)	11.9	9.0	35.8	44.8	36.3	0.0	63.7
Added content: None	FIFO	11.8	10.3	41.2	51.5	0.0	0.0	100.0
Added content: None	LRU (rule)	11.8	10.3	44.1	54.4	11.8	0.0	88.2
Added content: V	Generative	14.4	9.9	14.4	24.4	90.9	6.7	2.4
Added content: V	Pi + RP	14.9	6.0	3.0	9.0	100.0	0.0	0.0
Added content: V	Pi	15.2	10.6	12.1	22.7	97.5	2.5	0.0
Added content: V	Structured	14.9	6.0	11.9	17.9	93.5	5.0	1.5
Added content: V	TRACE	14.8	9.8	14.8	24.6	91.8	3.3	4.9
Added content: V	ACON-UTCO	14.7	11.8	17.6	29.4	92.6	5.9	1.5
Added content: V	ACON-UT	14.7	14.7	10.3	25.0	92.2	5.9	2.0
Added content: V	Hermes	11.9	10.4	31.3	41.8	68.7	23.9	7.5
Added content: V	Selective	15.1	10.1	34.7	44.7	17.9	0.0	82.1
Added content: V	LRU (LLM)	15.9	11.1	27.0	38.1	37.6	0.0	62.4
Added content: V	FIFO	14.7	7.4	44.1	51.5	0.0	0.0	100.0
Added content: V	LRU (rule)	14.7	11.8	32.4	44.1	17.6	0.0	82.4
Added content: I	Generative	13.5	9.0	27.1	36.1	69.7	13.9	16.3
Added content: I	Pi + RP	13.4	10.4	7.5	17.9	99.5	0.5	0.0
Added content: I	Pi	13.4	7.5	23.9	31.3	73.6	9.0	17.4
Added content: I	Structured	13.6	7.6	24.2	31.8	76.3	8.1	15.7
Added content: I	TRACE	14.1	7.8	23.4	31.3	70.3	14.6	15.1
Added content: I	ACON-UTCO	13.4	9.0	34.3	43.3	60.7	19.9	19.4
Added content: I	ACON-UT	13.6	10.6	34.8	45.5	64.1	20.7	15.2
Added content: I	Hermes	13.2	10.3	41.2	51.5	44.1	24.5	31.4
Added content: I	Selective	13.6	12.1	35.7	47.7	11.2	0.0	88.8
Added content: I	LRU (LLM)	14.3	12.7	28.6	41.3	22.8	0.0	77.2
Added content: I	FIFO	13.2	11.8	44.1	55.9	0.0	0.0	100.0
Added content: I	LRU (rule)	13.2	11.8	33.8	45.6	11.8	0.0	88.2
Added content: V+I	Generative	9.5	7.6	25.4	33.0	72.9	12.4	14.8
Added content: V+I	Pi + RP	9.1	7.6	6.1	13.6	100.0	0.0	0.0
Added content: V+I	Pi	9.0	6.0	17.9	23.9	83.6	8.5	8.0
Added content: V+I	Structured	9.1	6.1	22.7	28.8	82.3	7.1	10.6
Added content: V+I	TRACE	9.5	7.9	27.0	34.9	65.1	14.8	20.1
Added content: V+I	ACON-UTCO	9.4	7.8	32.8	40.6	64.1	17.2	18.8
Added content: V+I	ACON-UT	9.0	6.0	34.3	40.3	63.2	17.4	19.4
Added content: V+I	Hermes	11.8	11.8	36.8	48.5	52.0	21.6	26.5
Added content: V+I	Selective	9.1	8.6	39.9	48.5	15.8	0.0	84.2
Added content: V+I	LRU (LLM)	9.7	9.7	33.9	43.5	31.2	0.0	68.8
Added content: V+I	FIFO	8.8	7.4	47.1	54.4	0.0	0.0	100.0
Added content: V+I	LRU (rule)	8.8	8.8	38.2	47.1	17.6	0.0	82.4

violation@3; V=volume, I=compactor-targeted injection. Family pools remain separate from individual strategies.

ARCBench original results · Table 29

S5/S7 and strategy run-level rates

Read this configuration independently from the primary S1 comparison.

S5/S7 and strategy run-level rates
Group	Condition	Full violation %	Comp. violation %	Retained share %	Partial share %	Absent share %	Violation if retained %	Violation if partial %	Violation if absent %
S5: added content (Hermes, four models)	None	8.1	25.2	70.1	26.9	3.0	21.1	29.4	83.3
S5: added content (Hermes, four models)	V	8.2	23.1	71.9	22.9	5.2	16.3	37.0	57.1
S5: added content (Hermes, four models)	I	8.2	35.6	42.5	26.4	31.1	22.2	49.1	42.4
S5: added content (Hermes, four models)	V+I	8.2	27.9	54.5	23.4	22.1	17.4	41.5	39.3
S7: no attack, by defense (GLM)	None	6.9	15.7	96.1	3.9	0.0	13.3	75.0	–
S7: no attack, by defense (GLM)	Isolation	6.9	18.6	94.1	5.9	0.0	17.7	33.3	–
S7: no attack, by defense (GLM)	Instructional	6.9	14.7	97.1	2.9	0.0	13.1	66.7	–
S7: no attack, by defense (GLM)	Sandwich	6.9	15.7	93.1	6.9	0.0	13.7	42.9	–
S7: no attack, by defense (GLM)	Rule pres.	6.9	14.7	100.0	0.0	0.0	14.7	–	–
S7: escape, by defense (GLM)	None	6.9	30.4	66.7	13.7	19.6	26.5	28.6	45.0
S7: escape, by defense (GLM)	Isolation	6.9	21.6	85.3	6.9	7.8	23.0	0.0	25.0
S7: escape, by defense (GLM)	Instructional	6.9	21.6	87.3	7.8	4.9	20.2	0.0	80.0
S7: escape, by defense (GLM)	Sandwich	6.9	20.6	82.4	11.8	5.9	15.5	33.3	66.7
S7: escape, by defense (GLM)	Rule pres.	6.9	16.7	99.0	1.0	0.0	15.8	100.0	–
Strategy study: family by added content	Generative None	9.5	14.4	91.2	7.8	0.9	12.6	27.3	76.9
Strategy study: family by added content	Generative V	9.1	15.0	90.9	6.7	2.4	12.3	32.3	70.6
Strategy study: family by added content	Generative I	9.5	24.6	69.7	13.9	16.3	14.4	45.4	50.4
Strategy study: family by added content	Generative V+I	7.4	22.8	72.9	12.4	14.8	15.1	40.4	46.6
Strategy study: family by added content	Selective None	9.4	38.4	15.9	0.0	84.1	20.6	–	41.8
Strategy study: family by added content	Selective V	9.0	36.7	17.9	0.0	82.1	18.7	–	40.6
Strategy study: family by added content	Selective I	9.5	40.2	11.2	0.0	88.8	13.4	–	43.6
Strategy study: family by added content	Selective V+I	7.1	38.9	15.8	0.0	84.2	17.0	–	43.0

Per-run rates and rule-state shares; S7 uses GLM only. Not violation@3 or violation@6.

ARCBench original results · Table 30

S5–S7 behavior labels

Read this configuration independently from the primary S1 comparison.

S5–S7 behavior labels
Group	Condition	Approval %	Scope %	Unregistered %	Acknowledged %
S5: added content	None	45.9	5.4	27.0	21.6
S5: added content	V	40.0	6.7	26.7	26.7
S5: added content	I	60.7	7.1	17.9	14.3
S5: added content	V+I	42.3	11.5	38.5	7.7
S6: attack, no defense	None	100.0	0.0	0.0	0.0
S6: attack, no defense	Escape	61.5	15.4	15.4	7.7
S7: defense under escape	Isolation	90.9	0.0	9.1	0.0
S7: defense under escape	Instructional	83.3	0.0	16.7	0.0
S7: defense under escape	Sandwich	50.0	12.5	25.0	12.5
S7: defense under escape	Rule pres.	71.4	14.3	0.0	14.3

Readable-reasoning runs that retain the rule and violate only after compaction.

ARCBench original results · Table 31

Original publication

Original paper v1, 2 November 2026; reviewed 4 November. Primary S1 baseline, with model/channel, strategy, budget and attack-defense details separated.

Reviewed 2026-11-04

Evidence available for this benchmark

This source record contains 8 numeric observations. The source distinguishes 4 model identities or configurations. A model may appear more than once; check the version, tools and run budget before comparing. Read results and row-level evidence →

Reported metric: S1 violation@3 (at least one violation across three valid repetitions)
Recorded unit: percent
Metric direction: lower is better

Start with these resources

Catalog source · arxiv.org

Cited documents and dates (1)

ARCBench · original v1 evaluation · 2026-11-02

Evaluation context and evidence

Source: Model reports
Publisher: Not recorded
Benchmark release: 2026-11-02
Collected: Not recorded
Modality: Not recorded

These are source-reported observations, not independent reruns by this site. Check the task version, evaluation protocol, unit and reporting date before comparing models. Results from different setups are not a universal ranking.

Model	Value	Unit	Protocol	Reported date and basis	Source
GPT-Luna	6.1	percent	S1; Full history, no compaction; no injection or defense; personal cases c1/c2, organizational cases c3/c4; 33 valid task-channel settings; violation@3; same endpoint agent and compactor: ~openai/gpt-luna-latest; pi revision 97fa14e3; temperature 0.7, seed 7, four tool steps; source publication date, exact run dates unknown	2026-11-02 · document_publication	ARCBench · original v1 evaluation
DeepSeek-V4-Flash-0731	8.8	percent	S1; Full history, no compaction; no injection or defense; personal cases c1/c2, organizational cases c3/c4; 34 valid task-channel settings; violation@3; same endpoint agent and compactor: deepseek/deepseek-v4-flash-0731; pi revision 97fa14e3; temperature 0.7, seed 7, four tool steps; source publication date, exact run dates unknown	2026-11-02 · document_publication	ARCBench · original v1 evaluation
Qwen3.7-Flash	14.7	percent	S1; Full history, no compaction; no injection or defense; personal cases c1/c2, organizational cases c3/c4; 34 valid task-channel settings; violation@3; same endpoint agent and compactor: qwen/qwen3.7-flash; pi revision 97fa14e3; temperature 0.7, seed 7, four tool steps; source publication date, exact run dates unknown	2026-11-02 · document_publication	ARCBench · original v1 evaluation
GLM-5.3-Flash	8.8	percent	S1; Full history, no compaction; no injection or defense; personal cases c1/c2, organizational cases c3/c4; 34 valid task-channel settings; violation@3; same endpoint agent and compactor: z-ai/glm-5.3-flash; pi revision 97fa14e3; temperature 0.7, seed 7, four tool steps; source publication date, exact run dates unknown	2026-11-02 · document_publication	ARCBench · original v1 evaluation
GPT-Luna	33.3	percent	S1; Hermes compaction, no extra budget; no injection or defense; personal cases c1/c2, organizational cases c3/c4; 33 valid task-channel settings; violation@3; same endpoint agent and compactor: ~openai/gpt-luna-latest; pi revision 97fa14e3; temperature 0.7, seed 7, four tool steps; source publication date, exact run dates unknown	2026-11-02 · document_publication	ARCBench · original v1 evaluation
DeepSeek-V4-Flash-0731	26.5	percent	S1; Hermes compaction, no extra budget; no injection or defense; personal cases c1/c2, organizational cases c3/c4; 34 valid task-channel settings; violation@3; same endpoint agent and compactor: deepseek/deepseek-v4-flash-0731; pi revision 97fa14e3; temperature 0.7, seed 7, four tool steps; source publication date, exact run dates unknown	2026-11-02 · document_publication	ARCBench · original v1 evaluation
Qwen3.7-Flash	61.8	percent	S1; Hermes compaction, no extra budget; no injection or defense; personal cases c1/c2, organizational cases c3/c4; 34 valid task-channel settings; violation@3; same endpoint agent and compactor: qwen/qwen3.7-flash; pi revision 97fa14e3; temperature 0.7, seed 7, four tool steps; source publication date, exact run dates unknown	2026-11-02 · document_publication	ARCBench · original v1 evaluation
GLM-5.3-Flash	29.4	percent	S1; Hermes compaction, no extra budget; no injection or defense; personal cases c1/c2, organizational cases c3/c4; 34 valid task-channel settings; violation@3; same endpoint agent and compactor: z-ai/glm-5.3-flash; pi revision 97fa14e3; temperature 0.7, seed 7, four tool steps; source publication date, exact run dates unknown	2026-11-02 · document_publication	ARCBench · original v1 evaluation

Rules after a shortened history

Lower is better; cases are not runs

Keep the exact endpoint and harness

Retention is a different measurement

S1: full-history versus Hermes

S1 by model and delivery channel

S1 run-level outcomes

S1 behavior attribution counts

S3 strategy comparison

S3 strategies by model

S4 budget sweep

S2–S4 run-level rates and rule states

S2–S4 behavior labels

S5 added-content experiment

S6 attack constructions

S7 defenses with and without attack

Strategy and added-content combinations

S5/S7 and strategy run-level rates

S5–S7 behavior labels

Original publication

Evidence available for this benchmark

Start with these resources

Evaluation context and evidence

Reported results 8

Explore related benchmarks

Put this benchmark in context

Continue exploring