GraphNIAH

AI Benchmark Radar / Equality Machine · Reviewed 2026-11-02

Primary results: paper v1 Table 23 pooled question-depth accuracy. GPT-5.6-Sol uses 300 examples at depths 8/16/32; seven peers use 400 at 4/8/16/32. Full experimental tables follow.

Find citation paths, not isolated facts

Can a model follow a citation graph described inside a long prompt? GraphNIAH asks whether one paper reaches another, which candidate is reachable, or how many papers are reachable. Answers are calculated from the graph.

GraphNIAH paper v1 · GraphNIAH author project

A versioned citation-graph collection

Version 1 uses the Semantic Scholar 2026-10-06 snapshot and papers dated from 2026-04-01. Table 2 reports 1,760 distinct evaluation examples, with no training/development split. The author pipeline and released dataset support rebuilding later collections.

GraphNIAH paper v1 · GraphNIAH author dataset · GraphNIAH author code

Keep every experiment separate

Primary scores use only Table 23’s pooled question-depth column. This is one experiment, not an overall score across GraphNIAH. Each extracted final answer must exactly match graph ground truth. The 200k context label is a target, not the measured length of every prompt.

GraphNIAH paper v1

Model settings and incomplete subsets

Default temperature is 1.0 and reasoning effort is low. Qwen has a 40,000-token output cap and provider fallback. GPT-5.6-Sol lacks depth 4 and some component-size examples; its pooled rows therefore have smaller denominators.

GraphNIAH paper v1

What this result does not establish

Graph reachability is narrower than general research ability. Qwen’s undisclosed training cutoffs are approximated by release dates; this does not independently prove contamination absence. Publication dates identify this paper version, not individual evaluation runs.

GraphNIAH paper v1 · GraphNIAH author code

Question-depth experiments

Every distinct reported accuracy cell for this experiment is retained, including conditional subsets. Repeated copies of the same measurement are shown once. These rows do not replace the primary pooled question-depth metric.

Question-depth experiments · paper v1 accuracy (%) with 95% Wilson intervals
Model	Source table	Condition / effort	Accuracy [95% interval]	Evaluated examples N
GPT-5.6-Sol	T23	all three reachability tasks; question depth k=8 \| low	75.00 [65.70, 82.45]	100
GPT-5.6-Sol	T23	all three reachability tasks; question depth k=16 \| low	56.00 [46.23, 65.33]	100
GPT-5.6-Sol	T23	all three reachability tasks; question depth k=32 \| low	54.00 [44.26, 63.44]	100
GPT-5.6-Sol	T23	all three reachability tasks; question depth pooled k=8,16,32 only \| low	61.67† [56.05, 66.99]	300
GPT-5.6-Luna	T23	all three reachability tasks; question depth k=4 \| low	66.00 [56.28, 74.54]	100
GPT-5.6-Luna	T23	all three reachability tasks; question depth k=8 \| low	39.00 [30.02, 48.80]	100
GPT-5.6-Luna	T23	all three reachability tasks; question depth k=16 \| low	37.00 [28.18, 46.78]	100
GPT-5.6-Luna	T23	all three reachability tasks; question depth k=32 \| low	34.00 [25.46, 43.72]	100
GPT-5.6-Luna	T23	all three reachability tasks; question depth pooled k=4,8,16,32 \| low	44.00 [39.22, 48.90]	400
Gemini-3.1-Pro	T23	all three reachability tasks; question depth k=4 \| low	71.00 [61.46, 78.99]	100
Gemini-3.1-Pro	T23	all three reachability tasks; question depth k=8 \| low	62.00 [52.21, 70.90]	100
Gemini-3.1-Pro	T23	all three reachability tasks; question depth k=16 \| low	57.00 [47.22, 66.27]	100
Gemini-3.1-Pro	T23	all three reachability tasks; question depth k=32 \| low	52.00 [42.32, 61.54]	100
Gemini-3.1-Pro	T23	all three reachability tasks; question depth pooled k=4,8,16,32 \| low	60.50 [55.63, 65.17]	400
Gemini-3.8-Flash	T23	all three reachability tasks; question depth k=4 \| low	87.00 [79.02, 92.24]	100
Gemini-3.8-Flash	T23	all three reachability tasks; question depth k=8 \| low	83.00 [74.45, 89.11]	100
Gemini-3.8-Flash	T23	all three reachability tasks; question depth k=16 \| low	87.00 [79.02, 92.24]	100
Gemini-3.8-Flash	T23	all three reachability tasks; question depth k=32 \| low	59.00 [49.20, 68.13]	100
Gemini-3.8-Flash	T23	all three reachability tasks; question depth pooled k=4,8,16,32 \| low	79.00 [74.74, 82.71]	400
Gemini-3.6-Flash	T23	all three reachability tasks; question depth k=4 \| low	75.00 [65.70, 82.45]	100
Gemini-3.6-Flash	T23	all three reachability tasks; question depth k=8 \| low	56.00 [46.23, 65.33]	100
Gemini-3.6-Flash	T23	all three reachability tasks; question depth k=16 \| low	47.00 [37.51, 56.71]	100
Gemini-3.6-Flash	T23	all three reachability tasks; question depth k=32 \| low	41.00 [31.87, 50.80]	100
Gemini-3.6-Flash	T23	all three reachability tasks; question depth pooled k=4,8,16,32 \| low	54.75 [49.85, 59.56]	400
Qwen3.5-397B-A17B	T23	all three reachability tasks; question depth k=4 \| low	65.00 [55.25, 73.64]	100
Qwen3.5-397B-A17B	T23	all three reachability tasks; question depth k=8 \| low	48.00 [38.46, 57.68]	100
Qwen3.5-397B-A17B	T23	all three reachability tasks; question depth k=16 \| low	42.00 [32.80, 51.79]	100
Qwen3.5-397B-A17B	T23	all three reachability tasks; question depth k=32 \| low	38.00 [29.10, 47.79]	100
Qwen3.5-397B-A17B	T23	all three reachability tasks; question depth pooled k=4,8,16,32 \| low	48.25 [43.39, 53.14]	400
Qwen3.5-122B-A10B	T23	all three reachability tasks; question depth k=4 \| low	65.00 [55.25, 73.64]	100
Qwen3.5-122B-A10B	T23	all three reachability tasks; question depth k=8 \| low	45.00 [35.61, 54.76]	100
Qwen3.5-122B-A10B	T23	all three reachability tasks; question depth k=16 \| low	40.00 [30.94, 49.80]	100
Qwen3.5-122B-A10B	T23	all three reachability tasks; question depth k=32 \| low	37.00 [28.18, 46.78]	100
Qwen3.5-122B-A10B	T23	all three reachability tasks; question depth pooled k=4,8,16,32 \| low	46.75 [41.91, 51.65]	400
Qwen3.5-35B-A3B	T23	all three reachability tasks; question depth k=4 \| low	63.00 [53.22, 71.82]	100
Qwen3.5-35B-A3B	T23	all three reachability tasks; question depth k=8 \| low	42.00 [32.80, 51.79]	100
Qwen3.5-35B-A3B	T23	all three reachability tasks; question depth k=16 \| low	37.00 [28.18, 46.78]	100
Qwen3.5-35B-A3B	T23	all three reachability tasks; question depth k=32 \| low	39.00 [30.02, 48.80]	100
Qwen3.5-35B-A3B	T23	all three reachability tasks; question depth pooled k=4,8,16,32 \| low	45.25 [40.44, 50.15]	400
GPT-5.6-Sol	T5	YES/NO; question depth k=8 \| low	96.43 [82.29, 99.37]	28
GPT-5.6-Sol	T5	YES/NO; question depth k=16 \| low	57.50 [42.19, 71.49]	40
GPT-5.6-Sol	T5	YES/NO; question depth k=32 \| low	53.49 [38.92, 67.49]	43
GPT-5.6-Sol	T5	YES/NO; question depth pooled k=8,16,32 only \| low	65.77† [56.54, 73.93]	111
GPT-5.6-Luna	T5	YES/NO; question depth k=4 \| low	94.12 [80.91, 98.37]	34
GPT-5.6-Luna	T5	YES/NO; question depth k=8 \| low	50.00 [32.63, 67.37]	28
GPT-5.6-Luna	T5	YES/NO; question depth k=16 \| low	50.00 [35.20, 64.80]	40
GPT-5.6-Luna	T5	YES/NO; question depth k=32 \| low	51.16 [36.75, 65.38]	43
GPT-5.6-Luna	T5	YES/NO; question depth pooled k=4,8,16,32 \| low	60.69 [52.56, 68.27]	145
Gemini-3.1-Pro	T5	YES/NO; question depth k=4 \| low	100.00 [89.85, 100.00]	34
Gemini-3.1-Pro	T5	YES/NO; question depth k=8 \| low	89.29 [72.80, 96.29]	28
Gemini-3.1-Pro	T5	YES/NO; question depth k=16 \| low	67.50 [52.02, 79.92]	40
Gemini-3.1-Pro	T5	YES/NO; question depth k=32 \| low	67.44 [52.52, 79.51]	43
Gemini-3.1-Pro	T5	YES/NO; question depth pooled k=4,8,16,32 \| low	79.31 [72.00, 85.11]	145
Gemini-3.8-Flash	T5	YES/NO; question depth k=4 \| low	97.06 [85.08, 99.48]	34
Gemini-3.8-Flash	T5	YES/NO; question depth k=8 \| low	100.00 [87.94, 100.00]	28
Gemini-3.8-Flash	T5	YES/NO; question depth k=16 \| low	85.00 [70.93, 92.94]	40
Gemini-3.8-Flash	T5	YES/NO; question depth k=32 \| low	67.44 [52.52, 79.51]	43
Gemini-3.8-Flash	T5	YES/NO; question depth pooled k=4,8,16,32 \| low	85.52 [78.87, 90.33]	145
Gemini-3.6-Flash	T5	YES/NO; question depth k=4 \| low	100.00 [89.85, 100.00]	34
Gemini-3.6-Flash	T5	YES/NO; question depth k=8 \| low	82.14 [64.41, 92.12]	28
Gemini-3.6-Flash	T5	YES/NO; question depth k=16 \| low	67.50 [52.02, 79.92]	40
Gemini-3.6-Flash	T5	YES/NO; question depth k=32 \| low	58.14 [43.33, 71.62]	43
Gemini-3.6-Flash	T5	YES/NO; question depth pooled k=4,8,16,32 \| low	75.17 [67.55, 81.49]	145
Qwen3.5-397B-A17B	T5	YES/NO; question depth k=4 \| low	94.12 [80.91, 98.37]	34
Qwen3.5-397B-A17B	T5	YES/NO; question depth k=8 \| low	71.43 [52.94, 84.75]	28
Qwen3.5-397B-A17B	T5	YES/NO; question depth k=16 \| low	50.00 [35.20, 64.80]	40
Qwen3.5-397B-A17B	T5	YES/NO; question depth k=32 \| low	51.16 [36.75, 65.38]	43
Qwen3.5-397B-A17B	T5	YES/NO; question depth pooled k=4,8,16,32 \| low	64.83 [56.76, 72.13]	145
Qwen3.5-122B-A10B	T5	YES/NO; question depth k=4 \| low	94.12 [80.91, 98.37]	34
Qwen3.5-122B-A10B	T5	YES/NO; question depth k=8 \| low	60.71 [42.41, 76.43]	28
Qwen3.5-122B-A10B	T5	YES/NO; question depth k=16 \| low	50.00 [35.20, 64.80]	40
Qwen3.5-122B-A10B	T5	YES/NO; question depth k=32 \| low	51.16 [36.75, 65.38]	43
Qwen3.5-122B-A10B	T5	YES/NO; question depth pooled k=4,8,16,32 \| low	62.76 [54.66, 70.20]	145
Qwen3.5-35B-A3B	T5	YES/NO; question depth k=4 \| low	88.24 [73.38, 95.33]	34
Qwen3.5-35B-A3B	T5	YES/NO; question depth k=8 \| low	57.14 [39.07, 73.49]	28
Qwen3.5-35B-A3B	T5	YES/NO; question depth k=16 \| low	50.00 [35.20, 64.80]	40
Qwen3.5-35B-A3B	T5	YES/NO; question depth k=32 \| low	55.81 [41.11, 69.57]	43
Qwen3.5-35B-A3B	T5	YES/NO; question depth pooled k=4,8,16,32 \| low	62.07 [53.96, 69.56]	145
GPT-5.6-Sol	T6	multiple-choice; question depth k=8 \| low	100.00 [90.82, 100.00]	38
GPT-5.6-Sol	T6	multiple-choice; question depth k=16 \| low	86.21 [69.44, 94.50]	29
GPT-5.6-Sol	T6	multiple-choice; question depth k=32 \| low	69.23 [50.01, 83.50]	26
GPT-5.6-Sol	T6	multiple-choice; question depth pooled k=8,16,32 only \| low	87.10† [78.79, 92.46]	93
GPT-5.6-Luna	T6	multiple-choice; question depth k=4 \| low	100.00 [89.85, 100.00]	34
GPT-5.6-Luna	T6	multiple-choice; question depth k=8 \| low	65.79 [49.89, 78.79]	38
GPT-5.6-Luna	T6	multiple-choice; question depth k=16 \| low	58.62 [40.74, 74.49]	29
GPT-5.6-Luna	T6	multiple-choice; question depth k=32 \| low	46.15 [28.76, 64.54]	26
GPT-5.6-Luna	T6	multiple-choice; question depth pooled k=4,8,16,32 \| low	69.29 [60.80, 76.65]	127
Gemini-3.1-Pro	T6	multiple-choice; question depth k=4 \| low	100.00 [89.85, 100.00]	34
Gemini-3.1-Pro	T6	multiple-choice; question depth k=8 \| low	92.11 [79.20, 97.28]	38
Gemini-3.1-Pro	T6	multiple-choice; question depth k=16 \| low	89.66 [73.61, 96.42]	29
Gemini-3.1-Pro	T6	multiple-choice; question depth k=32 \| low	65.38 [46.22, 80.59]	26
Gemini-3.1-Pro	T6	multiple-choice; question depth pooled k=4,8,16,32 \| low	88.19 [81.43, 92.71]	127
Gemini-3.8-Flash	T6	multiple-choice; question depth k=4 \| low	100.00 [89.85, 100.00]	34
Gemini-3.8-Flash	T6	multiple-choice; question depth k=8 \| low	100.00 [90.82, 100.00]	38
Gemini-3.8-Flash	T6	multiple-choice; question depth k=16 \| low	100.00 [88.30, 100.00]	29
Gemini-3.8-Flash	T6	multiple-choice; question depth k=32 \| low	80.77 [62.12, 91.49]	26
Gemini-3.8-Flash	T6	multiple-choice; question depth pooled k=4,8,16,32 \| low	96.06 [91.11, 98.31]	127
Gemini-3.6-Flash	T6	multiple-choice; question depth k=4 \| low	100.00 [89.85, 100.00]	34
Gemini-3.6-Flash	T6	multiple-choice; question depth k=8 \| low	73.68 [57.99, 85.03]	38
Gemini-3.6-Flash	T6	multiple-choice; question depth k=16 \| low	62.07 [44.00, 77.31]	29
Gemini-3.6-Flash	T6	multiple-choice; question depth k=32 \| low	46.15 [28.76, 64.54]	26
Gemini-3.6-Flash	T6	multiple-choice; question depth pooled k=4,8,16,32 \| low	72.44 [64.10, 79.47]	127
Qwen3.5-397B-A17B	T6	multiple-choice; question depth k=4 \| low	97.06 [85.08, 99.48]	34
Qwen3.5-397B-A17B	T6	multiple-choice; question depth k=8 \| low	73.68 [57.99, 85.03]	38
Qwen3.5-397B-A17B	T6	multiple-choice; question depth k=16 \| low	75.86 [57.89, 87.78]	29
Qwen3.5-397B-A17B	T6	multiple-choice; question depth k=32 \| low	61.54 [42.53, 77.57]	26
Qwen3.5-397B-A17B	T6	multiple-choice; question depth pooled k=4,8,16,32 \| low	77.95 [69.98, 84.28]	127
Qwen3.5-122B-A10B	T6	multiple-choice; question depth k=4 \| low	97.06 [85.08, 99.48]	34
Qwen3.5-122B-A10B	T6	multiple-choice; question depth k=8 \| low	73.68 [57.99, 85.03]	38
Qwen3.5-122B-A10B	T6	multiple-choice; question depth k=16 \| low	68.97 [50.77, 82.72]	29
Qwen3.5-122B-A10B	T6	multiple-choice; question depth k=32 \| low	57.69 [38.95, 74.46]	26
Qwen3.5-122B-A10B	T6	multiple-choice; question depth pooled k=4,8,16,32 \| low	75.59 [67.44, 82.24]	127
Qwen3.5-35B-A3B	T6	multiple-choice; question depth k=4 \| low	97.06 [85.08, 99.48]	34
Qwen3.5-35B-A3B	T6	multiple-choice; question depth k=8 \| low	68.42 [52.54, 80.92]	38
Qwen3.5-35B-A3B	T6	multiple-choice; question depth k=16 \| low	58.62 [40.74, 74.49]	29
Qwen3.5-35B-A3B	T6	multiple-choice; question depth k=32 \| low	57.69 [38.95, 74.46]	26
Qwen3.5-35B-A3B	T6	multiple-choice; question depth pooled k=4,8,16,32 \| low	71.65 [63.27, 78.77]	127
GPT-5.6-Sol	T7	counting; question depth k=8 \| low	29.41 [16.83, 46.17]	34
GPT-5.6-Sol	T7	counting; question depth k=16 \| low	25.81 [13.70, 43.25]	31
GPT-5.6-Sol	T7	counting; question depth k=32 \| low	41.94 [26.42, 59.23]	31
GPT-5.6-Sol	T7	counting; question depth pooled k=8,16,32 only \| low	32.29† [23.78, 42.17]	96
GPT-5.6-Luna	T7	counting; question depth k=4 \| low	0.00 [0.00, 10.72]	32
GPT-5.6-Luna	T7	counting; question depth k=8 \| low	0.00 [0.00, 10.15]	34
GPT-5.6-Luna	T7	counting; question depth k=16 \| low	0.00 [0.00, 11.03]	31
GPT-5.6-Luna	T7	counting; question depth k=32 \| low	0.00 [0.00, 11.03]	31
GPT-5.6-Luna	T7	counting; question depth pooled k=4,8,16,32 \| low	0.00 [0.00, 2.91]	128
Gemini-3.1-Pro	T7	counting; question depth k=4 \| low	9.38 [3.24, 24.22]	32
Gemini-3.1-Pro	T7	counting; question depth k=8 \| low	5.88 [1.63, 19.09]	34
Gemini-3.1-Pro	T7	counting; question depth k=16 \| low	12.90 [5.13, 28.85]	31
Gemini-3.1-Pro	T7	counting; question depth k=32 \| low	19.35 [9.19, 36.28]	31
Gemini-3.1-Pro	T7	counting; question depth pooled k=4,8,16,32 \| low	11.72 [7.23, 18.44]	128
Gemini-3.8-Flash	T7	counting; question depth k=4 \| low	62.50 [45.25, 77.07]	32
Gemini-3.8-Flash	T7	counting; question depth k=8 \| low	50.00 [34.07, 65.93]	34
Gemini-3.8-Flash	T7	counting; question depth k=16 \| low	77.42 [60.19, 88.61]	31
Gemini-3.8-Flash	T7	counting; question depth k=32 \| low	29.03 [16.10, 46.59]	31
Gemini-3.8-Flash	T7	counting; question depth pooled k=4,8,16,32 \| low	54.69 [46.05, 63.05]	128
Gemini-3.6-Flash	T7	counting; question depth k=4 \| low	21.88 [11.02, 38.76]	32
Gemini-3.6-Flash	T7	counting; question depth k=8 \| low	14.71 [6.45, 30.13]	34
Gemini-3.6-Flash	T7	counting; question depth k=16 \| low	6.45 [1.79, 20.72]	31
Gemini-3.6-Flash	T7	counting; question depth k=32 \| low	12.90 [5.13, 28.85]	31
Gemini-3.6-Flash	T7	counting; question depth pooled k=4,8,16,32 \| low	14.06 [9.08, 21.14]	128
Qwen3.5-397B-A17B	T7	counting; question depth k=4 \| low	0.00 [0.00, 10.72]	32
Qwen3.5-397B-A17B	T7	counting; question depth k=8 \| low	0.00 [0.00, 10.15]	34
Qwen3.5-397B-A17B	T7	counting; question depth k=16 \| low	0.00 [0.00, 11.03]	31
Qwen3.5-397B-A17B	T7	counting; question depth k=32 \| low	0.00 [0.00, 11.03]	31
Qwen3.5-397B-A17B	T7	counting; question depth pooled k=4,8,16,32 \| low	0.00 [0.00, 2.91]	128
Qwen3.5-122B-A10B	T7	counting; question depth k=4 \| low	0.00 [0.00, 10.72]	32
Qwen3.5-122B-A10B	T7	counting; question depth k=8 \| low	0.00 [0.00, 10.15]	34
Qwen3.5-122B-A10B	T7	counting; question depth k=16 \| low	0.00 [0.00, 11.03]	31
Qwen3.5-122B-A10B	T7	counting; question depth k=32 \| low	0.00 [0.00, 11.03]	31
Qwen3.5-122B-A10B	T7	counting; question depth pooled k=4,8,16,32 \| low	0.00 [0.00, 2.91]	128
Qwen3.5-35B-A3B	T7	counting; question depth k=4 \| low	0.00 [0.00, 10.72]	32
Qwen3.5-35B-A3B	T7	counting; question depth k=8 \| low	0.00 [0.00, 10.15]	34
Qwen3.5-35B-A3B	T7	counting; question depth k=16 \| low	0.00 [0.00, 11.03]	31
Qwen3.5-35B-A3B	T7	counting; question depth k=32 \| low	0.00 [0.00, 11.03]	31
Qwen3.5-35B-A3B	T7	counting; question depth pooled k=4,8,16,32 \| low	0.00 [0.00, 2.91]	128
GPT-5.6-Sol	T11	YES/NO YES-label; question depth k=8 \| low	92.86 [68.53, 98.73]	14
GPT-5.6-Sol	T11	YES/NO YES-label; question depth k=16 \| low	15.00 [5.24, 36.04]	20
GPT-5.6-Sol	T11	YES/NO YES-label; question depth k=32 \| low	4.76 [0.85, 22.67]	21
GPT-5.6-Sol	T11	YES/NO YES-label; question depth pooled k=8,16,32 only \| low	30.91† [20.28, 44.03]	55
GPT-5.6-Sol	T11	YES/NO NO-label; question depth k=8 \| low	100.00 [78.47, 100.00]	14
GPT-5.6-Sol	T11	YES/NO NO-label; question depth k=16 \| low	100.00 [83.89, 100.00]	20
GPT-5.6-Sol	T11	YES/NO NO-label; question depth k=32 \| low	100.00 [85.13, 100.00]	22
GPT-5.6-Sol	T11	YES/NO NO-label; question depth pooled k=8,16,32 only \| low	100.00† [93.58, 100.00]	56
GPT-5.6-Luna	T11	YES/NO YES-label; question depth k=4 \| low	88.24 [65.66, 96.71]	17
GPT-5.6-Luna	T11	YES/NO YES-label; question depth k=8 \| low	7.14 [1.27, 31.47]	14
GPT-5.6-Luna	T11	YES/NO YES-label; question depth k=16 \| low	0.00 [0.00, 16.11]	20
GPT-5.6-Luna	T11	YES/NO YES-label; question depth k=32 \| low	0.00 [0.00, 15.46]	21
GPT-5.6-Luna	T11	YES/NO YES-label; question depth pooled k=4,8,16,32 \| low	22.22 [14.17, 33.09]	72
GPT-5.6-Luna	T11	YES/NO NO-label; question depth k=4 \| low	100.00 [81.57, 100.00]	17
GPT-5.6-Luna	T11	YES/NO NO-label; question depth k=8 \| low	92.86 [68.53, 98.73]	14
GPT-5.6-Luna	T11	YES/NO NO-label; question depth k=16 \| low	100.00 [83.89, 100.00]	20
GPT-5.6-Luna	T11	YES/NO NO-label; question depth k=32 \| low	100.00 [85.13, 100.00]	22
GPT-5.6-Luna	T11	YES/NO NO-label; question depth pooled k=4,8,16,32 \| low	98.63 [92.64, 99.76]	73
Gemini-3.1-Pro	T11	YES/NO YES-label; question depth k=4 \| low	100.00 [81.57, 100.00]	17
Gemini-3.1-Pro	T11	YES/NO YES-label; question depth k=8 \| low	78.57 [52.41, 92.43]	14
Gemini-3.1-Pro	T11	YES/NO YES-label; question depth k=16 \| low	35.00 [18.12, 56.71]	20
Gemini-3.1-Pro	T11	YES/NO YES-label; question depth k=32 \| low	33.33 [17.19, 54.63]	21
Gemini-3.1-Pro	T11	YES/NO YES-label; question depth pooled k=4,8,16,32 \| low	58.33 [46.81, 69.01]	72
Gemini-3.1-Pro	T11	YES/NO NO-label; question depth k=4 \| low	100.00 [81.57, 100.00]	17
Gemini-3.1-Pro	T11	YES/NO NO-label; question depth k=8 \| low	100.00 [78.47, 100.00]	14
Gemini-3.1-Pro	T11	YES/NO NO-label; question depth k=16 \| low	100.00 [83.89, 100.00]	20
Gemini-3.1-Pro	T11	YES/NO NO-label; question depth k=32 \| low	100.00 [85.13, 100.00]	22
Gemini-3.1-Pro	T11	YES/NO NO-label; question depth pooled k=4,8,16,32 \| low	100.00 [95.00, 100.00]	73
Gemini-3.8-Flash	T11	YES/NO YES-label; question depth k=4 \| low	100.00 [81.57, 100.00]	17
Gemini-3.8-Flash	T11	YES/NO YES-label; question depth k=8 \| low	100.00 [78.47, 100.00]	14
Gemini-3.8-Flash	T11	YES/NO YES-label; question depth k=16 \| low	90.00 [69.90, 97.21]	20
Gemini-3.8-Flash	T11	YES/NO YES-label; question depth k=32 \| low	100.00 [84.54, 100.00]	21
Gemini-3.8-Flash	T11	YES/NO YES-label; question depth pooled k=4,8,16,32 \| low	97.22 [90.43, 99.23]	72
Gemini-3.8-Flash	T11	YES/NO NO-label; question depth k=4 \| low	94.12 [73.02, 98.95]	17
Gemini-3.8-Flash	T11	YES/NO NO-label; question depth k=8 \| low	100.00 [78.47, 100.00]	14
Gemini-3.8-Flash	T11	YES/NO NO-label; question depth k=16 \| low	80.00 [58.40, 91.93]	20
Gemini-3.8-Flash	T11	YES/NO NO-label; question depth k=32 \| low	36.36 [19.73, 57.05]	22
Gemini-3.8-Flash	T11	YES/NO NO-label; question depth pooled k=4,8,16,32 \| low	73.97 [62.89, 82.66]	73
Gemini-3.6-Flash	T11	YES/NO YES-label; question depth k=4 \| low	100.00 [81.57, 100.00]	17
Gemini-3.6-Flash	T11	YES/NO YES-label; question depth k=8 \| low	64.29 [38.76, 83.66]	14
Gemini-3.6-Flash	T11	YES/NO YES-label; question depth k=16 \| low	35.00 [18.12, 56.71]	20
Gemini-3.6-Flash	T11	YES/NO YES-label; question depth k=32 \| low	19.05 [7.67, 40.00]	21
Gemini-3.6-Flash	T11	YES/NO YES-label; question depth pooled k=4,8,16,32 \| low	51.39 [40.07, 62.57]	72
Gemini-3.6-Flash	T11	YES/NO NO-label; question depth k=4 \| low	100.00 [81.57, 100.00]	17
Gemini-3.6-Flash	T11	YES/NO NO-label; question depth k=8 \| low	100.00 [78.47, 100.00]	14
Gemini-3.6-Flash	T11	YES/NO NO-label; question depth k=16 \| low	100.00 [83.89, 100.00]	20
Gemini-3.6-Flash	T11	YES/NO NO-label; question depth k=32 \| low	95.45 [78.20, 99.19]	22
Gemini-3.6-Flash	T11	YES/NO NO-label; question depth pooled k=4,8,16,32 \| low	98.63 [92.64, 99.76]	73
Qwen3.5-397B-A17B	T11	YES/NO YES-label; question depth k=4 \| low	88.24 [65.66, 96.71]	17
Qwen3.5-397B-A17B	T11	YES/NO YES-label; question depth k=8 \| low	42.86 [21.38, 67.41]	14
Qwen3.5-397B-A17B	T11	YES/NO YES-label; question depth k=16 \| low	0.00 [0.00, 16.11]	20
Qwen3.5-397B-A17B	T11	YES/NO YES-label; question depth k=32 \| low	0.00 [0.00, 15.46]	21
Qwen3.5-397B-A17B	T11	YES/NO YES-label; question depth pooled k=4,8,16,32 \| low	29.17 [19.94, 40.51]	72
Qwen3.5-397B-A17B	T11	YES/NO NO-label; question depth k=4 \| low	100.00 [81.57, 100.00]	17
Qwen3.5-397B-A17B	T11	YES/NO NO-label; question depth k=8 \| low	100.00 [78.47, 100.00]	14
Qwen3.5-397B-A17B	T11	YES/NO NO-label; question depth k=16 \| low	100.00 [83.89, 100.00]	20
Qwen3.5-397B-A17B	T11	YES/NO NO-label; question depth k=32 \| low	100.00 [85.13, 100.00]	22
Qwen3.5-397B-A17B	T11	YES/NO NO-label; question depth pooled k=4,8,16,32 \| low	100.00 [95.00, 100.00]	73
Qwen3.5-122B-A10B	T11	YES/NO YES-label; question depth k=4 \| low	88.24 [65.66, 96.71]	17
Qwen3.5-122B-A10B	T11	YES/NO YES-label; question depth k=8 \| low	21.43 [7.57, 47.59]	14
Qwen3.5-122B-A10B	T11	YES/NO YES-label; question depth k=16 \| low	0.00 [0.00, 16.11]	20
Qwen3.5-122B-A10B	T11	YES/NO YES-label; question depth k=32 \| low	0.00 [0.00, 15.46]	21
Qwen3.5-122B-A10B	T11	YES/NO YES-label; question depth pooled k=4,8,16,32 \| low	25.00 [16.44, 36.09]	72
Qwen3.5-122B-A10B	T11	YES/NO NO-label; question depth k=4 \| low	100.00 [81.57, 100.00]	17
Qwen3.5-122B-A10B	T11	YES/NO NO-label; question depth k=8 \| low	100.00 [78.47, 100.00]	14
Qwen3.5-122B-A10B	T11	YES/NO NO-label; question depth k=16 \| low	100.00 [83.89, 100.00]	20
Qwen3.5-122B-A10B	T11	YES/NO NO-label; question depth k=32 \| low	100.00 [85.13, 100.00]	22
Qwen3.5-122B-A10B	T11	YES/NO NO-label; question depth pooled k=4,8,16,32 \| low	100.00 [95.00, 100.00]	73
Qwen3.5-35B-A3B	T11	YES/NO YES-label; question depth k=4 \| low	76.47 [52.74, 90.45]	17
Qwen3.5-35B-A3B	T11	YES/NO YES-label; question depth k=8 \| low	14.29 [4.01, 39.94]	14
Qwen3.5-35B-A3B	T11	YES/NO YES-label; question depth k=16 \| low	0.00 [0.00, 16.11]	20
Qwen3.5-35B-A3B	T11	YES/NO YES-label; question depth k=32 \| low	9.52 [2.65, 28.91]	21
Qwen3.5-35B-A3B	T11	YES/NO YES-label; question depth pooled k=4,8,16,32 \| low	23.61 [15.30, 34.60]	72
Qwen3.5-35B-A3B	T11	YES/NO NO-label; question depth k=4 \| low	100.00 [81.57, 100.00]	17
Qwen3.5-35B-A3B	T11	YES/NO NO-label; question depth k=8 \| low	100.00 [78.47, 100.00]	14
Qwen3.5-35B-A3B	T11	YES/NO NO-label; question depth k=16 \| low	100.00 [83.89, 100.00]	20
Qwen3.5-35B-A3B	T11	YES/NO NO-label; question depth k=32 \| low	100.00 [85.13, 100.00]	22
Qwen3.5-35B-A3B	T11	YES/NO NO-label; question depth pooled k=4,8,16,32 \| low	100.00 [95.00, 100.00]	73

Default temperature 1.0; reasoning effort low unless stated. Context target 200k unless varied; other variables sampled as paper §4.1. Qwen uses at most 40000 output tokens and provider fallback. † denotes the publisher’s reduced subset. Compare identical conditions and denominators.

GraphNIAH v1 · Table 5 · GraphNIAH v1 · Table 6 · GraphNIAH v1 · Table 7 · GraphNIAH v1 · Table 11 · GraphNIAH v1 · Table 23

Component-size experiments

Component-size experiments · paper v1 accuracy (%) with 95% Wilson intervals
Model	Source table	Condition / effort	Accuracy [95% interval]	Evaluated examples N
GPT-5.6-Sol	T24	all three reachability tasks; component size 100 \| low; reduced answered subset (†)	77.78† [59.24, 89.39]	27
GPT-5.6-Sol	T24	all three reachability tasks; component size 150 \| low; reduced answered subset (†)	61.54† [42.53, 77.57]	26
GPT-5.6-Sol	T24	all three reachability tasks; component size 200 \| low	60.00 [50.20, 69.06]	100
GPT-5.6-Sol	T24	all three reachability tasks; component size 250 \| low	49.00 [39.42, 58.65]	100
GPT-5.6-Sol	T24	all three reachability tasks; component size pooled 100,150,200,250 \| low; reduced answered subset (†)	57.71† [51.55, 63.64]	253
GPT-5.6-Luna	T24	all three reachability tasks; component size 100 \| low	44.00 [34.67, 53.77]	100
GPT-5.6-Luna	T24	all three reachability tasks; component size 150 \| low	38.00 [29.10, 47.79]	100
GPT-5.6-Luna	T24	all three reachability tasks; component size 200 \| low	42.00 [32.80, 51.79]	100
GPT-5.6-Luna	T24	all three reachability tasks; component size 250 \| low	37.00 [28.18, 46.78]	100
GPT-5.6-Luna	T24	all three reachability tasks; component size pooled 100,150,200,250 \| low	40.25 [35.56, 45.13]	400
Gemini-3.1-Pro	T24	all three reachability tasks; component size 100 \| low	74.00 [64.63, 81.60]	100
Gemini-3.1-Pro	T24	all three reachability tasks; component size 150 \| low	57.00 [47.22, 66.27]	100
Gemini-3.1-Pro	T24	all three reachability tasks; component size 200 \| low	55.00 [45.24, 64.39]	100
Gemini-3.1-Pro	T24	all three reachability tasks; component size 250 \| low	46.00 [36.56, 55.74]	100
Gemini-3.1-Pro	T24	all three reachability tasks; component size pooled 100,150,200,250 \| low	58.00 [53.11, 62.74]	400
Gemini-3.8-Flash	T24	all three reachability tasks; component size 100 \| low	88.00 [80.19, 93.00]	100
Gemini-3.8-Flash	T24	all three reachability tasks; component size 150 \| low	77.00 [67.85, 84.16]	100
Gemini-3.8-Flash	T24	all three reachability tasks; component size 200 \| low	76.00 [66.77, 83.31]	100
Gemini-3.8-Flash	T24	all three reachability tasks; component size 250 \| low	67.00 [57.31, 75.44]	100
Gemini-3.8-Flash	T24	all three reachability tasks; component size pooled 100,150,200,250 \| low	77.00 [72.63, 80.86]	400
Gemini-3.6-Flash	T24	all three reachability tasks; component size 100 \| low	69.00 [59.37, 77.22]	100
Gemini-3.6-Flash	T24	all three reachability tasks; component size 150 \| low	52.00 [42.32, 61.54]	100
Gemini-3.6-Flash	T24	all three reachability tasks; component size 200 \| low	54.00 [44.26, 63.44]	100
Gemini-3.6-Flash	T24	all three reachability tasks; component size 250 \| low	43.00 [33.73, 52.78]	100
Gemini-3.6-Flash	T24	all three reachability tasks; component size pooled 100,150,200,250 \| low	54.50 [49.60, 59.31]	400
Qwen3.5-397B-A17B	T24	all three reachability tasks; component size 100 \| low	54.00 [44.26, 63.44]	100
Qwen3.5-397B-A17B	T24	all three reachability tasks; component size 150 \| low	40.00 [30.94, 49.80]	100
Qwen3.5-397B-A17B	T24	all three reachability tasks; component size 200 \| low	43.00 [33.73, 52.78]	100
Qwen3.5-397B-A17B	T24	all three reachability tasks; component size 250 \| low	40.00 [30.94, 49.80]	100
Qwen3.5-397B-A17B	T24	all three reachability tasks; component size pooled 100,150,200,250 \| low	44.25 [39.46, 49.15]	400
Qwen3.5-122B-A10B	T24	all three reachability tasks; component size 100 \| low	50.00 [40.38, 59.62]	100
Qwen3.5-122B-A10B	T24	all three reachability tasks; component size 150 \| low	40.00 [30.94, 49.80]	100
Qwen3.5-122B-A10B	T24	all three reachability tasks; component size 200 \| low	41.00 [31.87, 50.80]	100
Qwen3.5-122B-A10B	T24	all three reachability tasks; component size 250 \| low	36.00 [27.27, 45.76]	100
Qwen3.5-122B-A10B	T24	all three reachability tasks; component size pooled 100,150,200,250 \| low	41.75 [37.02, 46.64]	400
Qwen3.5-35B-A3B	T24	all three reachability tasks; component size 100 \| low	43.00 [33.73, 52.78]	100
Qwen3.5-35B-A3B	T24	all three reachability tasks; component size 150 \| low	38.00 [29.10, 47.79]	100
Qwen3.5-35B-A3B	T24	all three reachability tasks; component size 200 \| low	42.00 [32.80, 51.79]	100
Qwen3.5-35B-A3B	T24	all three reachability tasks; component size 250 \| low	37.00 [28.18, 46.78]	100
Qwen3.5-35B-A3B	T24	all three reachability tasks; component size pooled 100,150,200,250 \| low	40.00 [35.32, 44.87]	400
GPT-5.6-Sol	T8	YES/NO; component size 100 \| low; reduced answered subset (†)	75.00† [46.77, 91.11]	12
GPT-5.6-Sol	T8	YES/NO; component size 150 \| low; reduced answered subset (†)	57.14† [25.05, 84.18]	7
GPT-5.6-Sol	T8	YES/NO; component size 200 \| low	76.19 [61.47, 86.52]	42
GPT-5.6-Sol	T8	YES/NO; component size 250 \| low	71.43 [52.94, 84.75]	28
GPT-5.6-Sol	T8	YES/NO; component size pooled 100,150,200,250 \| low; reduced answered subset (†)	73.03† [63.00, 81.16]	89
GPT-5.6-Luna	T8	YES/NO; component size 100 \| low	65.79 [49.89, 78.79]	38
GPT-5.6-Luna	T8	YES/NO; component size 150 \| low	57.14 [39.07, 73.49]	28
GPT-5.6-Luna	T8	YES/NO; component size 200 \| low	59.52 [44.49, 72.96]	42
GPT-5.6-Luna	T8	YES/NO; component size 250 \| low	53.57 [35.81, 70.47]	28
GPT-5.6-Luna	T8	YES/NO; component size pooled 100,150,200,250 \| low	59.56 [51.16, 67.43]	136
Gemini-3.1-Pro	T8	YES/NO; component size 100 \| low	86.84 [72.67, 94.25]	38
Gemini-3.1-Pro	T8	YES/NO; component size 150 \| low	82.14 [64.41, 92.12]	28
Gemini-3.1-Pro	T8	YES/NO; component size 200 \| low	78.57 [64.06, 88.29]	42
Gemini-3.1-Pro	T8	YES/NO; component size 250 \| low	64.29 [45.83, 79.29]	28
Gemini-3.1-Pro	T8	YES/NO; component size pooled 100,150,200,250 \| low	78.68 [71.05, 84.72]	136
Gemini-3.8-Flash	T8	YES/NO; component size 100 \| low	86.84 [72.67, 94.25]	38
Gemini-3.8-Flash	T8	YES/NO; component size 150 \| low	85.71 [68.51, 94.30]	28
Gemini-3.8-Flash	T8	YES/NO; component size 200 \| low	78.57 [64.06, 88.29]	42
Gemini-3.8-Flash	T8	YES/NO; component size 250 \| low	89.29 [72.80, 96.29]	28
Gemini-3.8-Flash	T8	YES/NO; component size pooled 100,150,200,250 \| low	84.56 [77.55, 89.67]	136
Gemini-3.6-Flash	T8	YES/NO; component size 100 \| low	71.05 [55.24, 83.00]	38
Gemini-3.6-Flash	T8	YES/NO; component size 150 \| low	71.43 [52.94, 84.75]	28
Gemini-3.6-Flash	T8	YES/NO; component size 200 \| low	76.19 [61.47, 86.52]	42
Gemini-3.6-Flash	T8	YES/NO; component size 250 \| low	64.29 [45.83, 79.29]	28
Gemini-3.6-Flash	T8	YES/NO; component size pooled 100,150,200,250 \| low	71.32 [63.22, 78.26]	136
Qwen3.5-397B-A17B	T8	YES/NO; component size 100 \| low	73.68 [57.99, 85.03]	38
Qwen3.5-397B-A17B	T8	YES/NO; component size 150 \| low	57.14 [39.07, 73.49]	28
Qwen3.5-397B-A17B	T8	YES/NO; component size 200 \| low	61.90 [46.81, 75.00]	42
Qwen3.5-397B-A17B	T8	YES/NO; component size 250 \| low	57.14 [39.07, 73.49]	28
Qwen3.5-397B-A17B	T8	YES/NO; component size pooled 100,150,200,250 \| low	63.24 [54.87, 70.87]	136
Qwen3.5-122B-A10B	T8	YES/NO; component size 100 \| low	65.79 [49.89, 78.79]	38
Qwen3.5-122B-A10B	T8	YES/NO; component size 150 \| low	57.14 [39.07, 73.49]	28
Qwen3.5-122B-A10B	T8	YES/NO; component size 200 \| low	57.14 [42.21, 70.88]	42
Qwen3.5-122B-A10B	T8	YES/NO; component size 250 \| low	53.57 [35.81, 70.47]	28
Qwen3.5-122B-A10B	T8	YES/NO; component size pooled 100,150,200,250 \| low	58.82 [50.42, 66.74]	136
Qwen3.5-35B-A3B	T8	YES/NO; component size 100 \| low	60.53 [44.72, 74.40]	38
Qwen3.5-35B-A3B	T8	YES/NO; component size 150 \| low	57.14 [39.07, 73.49]	28
Qwen3.5-35B-A3B	T8	YES/NO; component size 200 \| low	57.14 [42.21, 70.88]	42
Qwen3.5-35B-A3B	T8	YES/NO; component size 250 \| low	53.57 [35.81, 70.47]	28
Qwen3.5-35B-A3B	T8	YES/NO; component size pooled 100,150,200,250 \| low	57.35 [48.95, 65.35]	136
GPT-5.6-Sol	T9	multiple-choice; component size 100 \| low; reduced answered subset (†)	87.50† [52.91, 97.76]	8
GPT-5.6-Sol	T9	multiple-choice; component size 150 \| low; reduced answered subset (†)	90.91† [62.26, 98.38]	11
GPT-5.6-Sol	T9	multiple-choice; component size 200 \| low	78.57 [60.46, 89.79]	28
GPT-5.6-Sol	T9	multiple-choice; component size 250 \| low	84.38 [68.25, 93.14]	32
GPT-5.6-Sol	T9	multiple-choice; component size pooled 100,150,200,250 \| low; reduced answered subset (†)	83.54† [73.85, 90.12]	79
GPT-5.6-Luna	T9	multiple-choice; component size 100 \| low	57.58 [40.81, 72.76]	33
GPT-5.6-Luna	T9	multiple-choice; component size 150 \| low	64.71 [47.91, 78.51]	34
GPT-5.6-Luna	T9	multiple-choice; component size 200 \| low	60.71 [42.41, 76.43]	28
GPT-5.6-Luna	T9	multiple-choice; component size 250 \| low	68.75 [51.43, 82.05]	32
GPT-5.6-Luna	T9	multiple-choice; component size pooled 100,150,200,250 \| low	62.99 [54.33, 70.89]	127
Gemini-3.1-Pro	T9	multiple-choice; component size 100 \| low	96.97 [84.68, 99.46]	33
Gemini-3.1-Pro	T9	multiple-choice; component size 150 \| low	91.18 [77.04, 96.95]	34
Gemini-3.1-Pro	T9	multiple-choice; component size 200 \| low	78.57 [60.46, 89.79]	28
Gemini-3.1-Pro	T9	multiple-choice; component size 250 \| low	84.38 [68.25, 93.14]	32
Gemini-3.1-Pro	T9	multiple-choice; component size pooled 100,150,200,250 \| low	88.19 [81.43, 92.71]	127
Gemini-3.8-Flash	T9	multiple-choice; component size 100 \| low	96.97 [84.68, 99.46]	33
Gemini-3.8-Flash	T9	multiple-choice; component size 150 \| low	94.12 [80.91, 98.37]	34
Gemini-3.8-Flash	T9	multiple-choice; component size 200 \| low	100.00 [87.94, 100.00]	28
Gemini-3.8-Flash	T9	multiple-choice; component size 250 \| low	100.00 [89.28, 100.00]	32
Gemini-3.8-Flash	T9	multiple-choice; component size pooled 100,150,200,250 \| low	97.64 [93.28, 99.19]	127
Gemini-3.6-Flash	T9	multiple-choice; component size 100 \| low	81.82 [65.61, 91.39]	33
Gemini-3.6-Flash	T9	multiple-choice; component size 150 \| low	76.47 [60.00, 87.56]	34
Gemini-3.6-Flash	T9	multiple-choice; component size 200 \| low	60.71 [42.41, 76.43]	28
Gemini-3.6-Flash	T9	multiple-choice; component size 250 \| low	78.12 [61.24, 88.98]	32
Gemini-3.6-Flash	T9	multiple-choice; component size pooled 100,150,200,250 \| low	74.80 [66.60, 81.55]	127
Qwen3.5-397B-A17B	T9	multiple-choice; component size 100 \| low	78.79 [62.25, 89.32]	33
Qwen3.5-397B-A17B	T9	multiple-choice; component size 150 \| low	70.59 [53.83, 83.17]	34
Qwen3.5-397B-A17B	T9	multiple-choice; component size 200 \| low	60.71 [42.41, 76.43]	28
Qwen3.5-397B-A17B	T9	multiple-choice; component size 250 \| low	75.00 [57.89, 86.75]	32
Qwen3.5-397B-A17B	T9	multiple-choice; component size pooled 100,150,200,250 \| low	71.65 [63.27, 78.77]	127
Qwen3.5-122B-A10B	T9	multiple-choice; component size 100 \| low	75.76 [58.98, 87.17]	33
Qwen3.5-122B-A10B	T9	multiple-choice; component size 150 \| low	70.59 [53.83, 83.17]	34
Qwen3.5-122B-A10B	T9	multiple-choice; component size 200 \| low	60.71 [42.41, 76.43]	28
Qwen3.5-122B-A10B	T9	multiple-choice; component size 250 \| low	65.62 [48.31, 79.59]	32
Qwen3.5-122B-A10B	T9	multiple-choice; component size pooled 100,150,200,250 \| low	68.50 [59.98, 75.94]	127
Qwen3.5-35B-A3B	T9	multiple-choice; component size 100 \| low	60.61 [43.68, 75.32]	33
Qwen3.5-35B-A3B	T9	multiple-choice; component size 150 \| low	64.71 [47.91, 78.51]	34
Qwen3.5-35B-A3B	T9	multiple-choice; component size 200 \| low	64.29 [45.83, 79.29]	28
Qwen3.5-35B-A3B	T9	multiple-choice; component size 250 \| low	68.75 [51.43, 82.05]	32
Qwen3.5-35B-A3B	T9	multiple-choice; component size pooled 100,150,200,250 \| low	64.57 [55.93, 72.35]	127
GPT-5.6-Sol	T10	counting; component size 100 \| low; reduced answered subset (†)	71.43† [35.89, 91.78]	7
GPT-5.6-Sol	T10	counting; component size 150 \| low; reduced answered subset (†)	25.00† [7.15, 59.07]	8
GPT-5.6-Sol	T10	counting; component size 200 \| low	20.00 [9.50, 37.31]	30
GPT-5.6-Sol	T10	counting; component size 250 \| low	5.00 [1.38, 16.50]	40
GPT-5.6-Sol	T10	counting; component size pooled 100,150,200,250 \| low; reduced answered subset (†)	17.65† [11.00, 27.10]	85
GPT-5.6-Luna	T10	counting; component size 100 \| low	0.00 [0.00, 11.70]	29
GPT-5.6-Luna	T10	counting; component size 150 \| low	0.00 [0.00, 9.18]	38
GPT-5.6-Luna	T10	counting; component size 200 \| low	0.00 [0.00, 11.35]	30
GPT-5.6-Luna	T10	counting; component size 250 \| low	0.00 [0.00, 8.76]	40
GPT-5.6-Luna	T10	counting; component size pooled 100,150,200,250 \| low	0.00 [0.00, 2.73]	137
Gemini-3.1-Pro	T10	counting; component size 100 \| low	31.03 [17.28, 49.23]	29
Gemini-3.1-Pro	T10	counting; component size 150 \| low	7.89 [2.72, 20.80]	38
Gemini-3.1-Pro	T10	counting; component size 200 \| low	0.00 [0.00, 11.35]	30
Gemini-3.1-Pro	T10	counting; component size 250 \| low	2.50 [0.44, 12.88]	40
Gemini-3.1-Pro	T10	counting; component size pooled 100,150,200,250 \| low	9.49 [5.63, 15.56]	137
Gemini-3.8-Flash	T10	counting; component size 100 \| low	79.31 [61.61, 90.15]	29
Gemini-3.8-Flash	T10	counting; component size 150 \| low	55.26 [39.71, 69.85]	38
Gemini-3.8-Flash	T10	counting; component size 200 \| low	50.00 [33.15, 66.85]	30
Gemini-3.8-Flash	T10	counting; component size 250 \| low	25.00 [14.19, 40.19]	40
Gemini-3.8-Flash	T10	counting; component size pooled 100,150,200,250 \| low	50.36 [42.10, 58.61]	137
Gemini-3.6-Flash	T10	counting; component size 100 \| low	51.72 [34.43, 68.61]	29
Gemini-3.6-Flash	T10	counting; component size 150 \| low	15.79 [7.44, 30.42]	38
Gemini-3.6-Flash	T10	counting; component size 200 \| low	16.67 [7.34, 33.56]	30
Gemini-3.6-Flash	T10	counting; component size 250 \| low	0.00 [0.00, 8.76]	40
Gemini-3.6-Flash	T10	counting; component size pooled 100,150,200,250 \| low	18.98 [13.29, 26.36]	137
Qwen3.5-397B-A17B	T10	counting; component size 100 \| low	0.00 [0.00, 11.70]	29
Qwen3.5-397B-A17B	T10	counting; component size 150 \| low	0.00 [0.00, 9.18]	38
Qwen3.5-397B-A17B	T10	counting; component size 200 \| low	0.00 [0.00, 11.35]	30
Qwen3.5-397B-A17B	T10	counting; component size 250 \| low	0.00 [0.00, 8.76]	40
Qwen3.5-397B-A17B	T10	counting; component size pooled 100,150,200,250 \| low	0.00 [0.00, 2.73]	137
Qwen3.5-122B-A10B	T10	counting; component size 100 \| low	0.00 [0.00, 11.70]	29
Qwen3.5-122B-A10B	T10	counting; component size 150 \| low	0.00 [0.00, 9.18]	38
Qwen3.5-122B-A10B	T10	counting; component size 200 \| low	0.00 [0.00, 11.35]	30
Qwen3.5-122B-A10B	T10	counting; component size 250 \| low	0.00 [0.00, 8.76]	40
Qwen3.5-122B-A10B	T10	counting; component size pooled 100,150,200,250 \| low	0.00 [0.00, 2.73]	137
Qwen3.5-35B-A3B	T10	counting; component size 100 \| low	0.00 [0.00, 11.70]	29
Qwen3.5-35B-A3B	T10	counting; component size 150 \| low	0.00 [0.00, 9.18]	38
Qwen3.5-35B-A3B	T10	counting; component size 200 \| low	0.00 [0.00, 11.35]	30
Qwen3.5-35B-A3B	T10	counting; component size 250 \| low	0.00 [0.00, 8.76]	40
Qwen3.5-35B-A3B	T10	counting; component size pooled 100,150,200,250 \| low	0.00 [0.00, 2.73]	137
GPT-5.6-Sol	T12	YES/NO YES-label; component size 100 \| low; reduced answered subset (†)	50.00† [18.76, 81.24]	6
GPT-5.6-Sol	T12	YES/NO YES-label; component size 150 \| low; reduced answered subset (†)	25.00† [4.56, 69.94]	4
GPT-5.6-Sol	T12	YES/NO YES-label; component size 200 \| low	52.38 [32.37, 71.66]	21
GPT-5.6-Sol	T12	YES/NO YES-label; component size 250 \| low	42.86 [21.38, 67.41]	14
GPT-5.6-Sol	T12	YES/NO YES-label; component size pooled 100,150,200,250 \| low; reduced answered subset (†)	46.67† [32.93, 60.92]	45
GPT-5.6-Sol	T12	YES/NO NO-label; component size 100 \| low; reduced answered subset (†)	100.00† [60.97, 100.00]	6
GPT-5.6-Sol	T12	YES/NO NO-label; component size 150 \| low; reduced answered subset (†)	100.00† [43.85, 100.00]	3
GPT-5.6-Sol	T12	YES/NO NO-label; component size 200 \| low	100.00 [84.54, 100.00]	21
GPT-5.6-Sol	T12	YES/NO NO-label; component size 250 \| low	100.00 [78.47, 100.00]	14
GPT-5.6-Sol	T12	YES/NO NO-label; component size pooled 100,150,200,250 \| low; reduced answered subset (†)	100.00† [91.97, 100.00]	44
GPT-5.6-Luna	T12	YES/NO YES-label; component size 100 \| low	31.58 [15.36, 53.99]	19
GPT-5.6-Luna	T12	YES/NO YES-label; component size 150 \| low	14.29 [4.01, 39.94]	14
GPT-5.6-Luna	T12	YES/NO YES-label; component size 200 \| low	19.05 [7.67, 40.00]	21
GPT-5.6-Luna	T12	YES/NO YES-label; component size 250 \| low	7.14 [1.27, 31.47]	14
GPT-5.6-Luna	T12	YES/NO YES-label; component size pooled 100,150,200,250 \| low	19.12 [11.53, 30.01]	68
GPT-5.6-Luna	T12	YES/NO NO-label; component size 100 \| low	100.00 [83.18, 100.00]	19
GPT-5.6-Luna	T12	YES/NO NO-label; component size 150 \| low	100.00 [78.47, 100.00]	14
GPT-5.6-Luna	T12	YES/NO NO-label; component size 200 \| low	100.00 [84.54, 100.00]	21
GPT-5.6-Luna	T12	YES/NO NO-label; component size 250 \| low	100.00 [78.47, 100.00]	14
GPT-5.6-Luna	T12	YES/NO NO-label; component size pooled 100,150,200,250 \| low	100.00 [94.65, 100.00]	68
Gemini-3.1-Pro	T12	YES/NO YES-label; component size 100 \| low	73.68 [51.21, 88.19]	19
Gemini-3.1-Pro	T12	YES/NO YES-label; component size 150 \| low	64.29 [38.76, 83.66]	14
Gemini-3.1-Pro	T12	YES/NO YES-label; component size 200 \| low	57.14 [36.55, 75.53]	21
Gemini-3.1-Pro	T12	YES/NO YES-label; component size 250 \| low	28.57 [11.72, 54.65]	14
Gemini-3.1-Pro	T12	YES/NO YES-label; component size pooled 100,150,200,250 \| low	57.35 [45.52, 68.40]	68
Gemini-3.1-Pro	T12	YES/NO NO-label; component size 100 \| low	100.00 [83.18, 100.00]	19
Gemini-3.1-Pro	T12	YES/NO NO-label; component size 150 \| low	100.00 [78.47, 100.00]	14
Gemini-3.1-Pro	T12	YES/NO NO-label; component size 200 \| low	100.00 [84.54, 100.00]	21
Gemini-3.1-Pro	T12	YES/NO NO-label; component size 250 \| low	100.00 [78.47, 100.00]	14
Gemini-3.1-Pro	T12	YES/NO NO-label; component size pooled 100,150,200,250 \| low	100.00 [94.65, 100.00]	68
Gemini-3.8-Flash	T12	YES/NO YES-label; component size 100 \| low	100.00 [83.18, 100.00]	19
Gemini-3.8-Flash	T12	YES/NO YES-label; component size 150 \| low	92.86 [68.53, 98.73]	14
Gemini-3.8-Flash	T12	YES/NO YES-label; component size 200 \| low	100.00 [84.54, 100.00]	21
Gemini-3.8-Flash	T12	YES/NO YES-label; component size 250 \| low	100.00 [78.47, 100.00]	14
Gemini-3.8-Flash	T12	YES/NO YES-label; component size pooled 100,150,200,250 \| low	98.53 [92.13, 99.74]	68
Gemini-3.8-Flash	T12	YES/NO NO-label; component size 100 \| low	73.68 [51.21, 88.19]	19
Gemini-3.8-Flash	T12	YES/NO NO-label; component size 150 \| low	78.57 [52.41, 92.43]	14
Gemini-3.8-Flash	T12	YES/NO NO-label; component size 200 \| low	57.14 [36.55, 75.53]	21
Gemini-3.8-Flash	T12	YES/NO NO-label; component size 250 \| low	78.57 [52.41, 92.43]	14
Gemini-3.8-Flash	T12	YES/NO NO-label; component size pooled 100,150,200,250 \| low	70.59 [58.89, 80.08]	68
Gemini-3.6-Flash	T12	YES/NO YES-label; component size 100 \| low	57.89 [36.28, 76.86]	19
Gemini-3.6-Flash	T12	YES/NO YES-label; component size 150 \| low	42.86 [21.38, 67.41]	14
Gemini-3.6-Flash	T12	YES/NO YES-label; component size 200 \| low	52.38 [32.37, 71.66]	21
Gemini-3.6-Flash	T12	YES/NO YES-label; component size 250 \| low	28.57 [11.72, 54.65]	14
Gemini-3.6-Flash	T12	YES/NO YES-label; component size pooled 100,150,200,250 \| low	47.06 [35.67, 58.76]	68
Gemini-3.6-Flash	T12	YES/NO NO-label; component size 100 \| low	84.21 [62.43, 94.48]	19
Gemini-3.6-Flash	T12	YES/NO NO-label; component size 150 \| low	100.00 [78.47, 100.00]	14
Gemini-3.6-Flash	T12	YES/NO NO-label; component size 200 \| low	100.00 [84.54, 100.00]	21
Gemini-3.6-Flash	T12	YES/NO NO-label; component size 250 \| low	100.00 [78.47, 100.00]	14
Gemini-3.6-Flash	T12	YES/NO NO-label; component size pooled 100,150,200,250 \| low	95.59 [87.81, 98.49]	68
Qwen3.5-397B-A17B	T12	YES/NO YES-label; component size 100 \| low	47.37 [27.33, 68.29]	19
Qwen3.5-397B-A17B	T12	YES/NO YES-label; component size 150 \| low	14.29 [4.01, 39.94]	14
Qwen3.5-397B-A17B	T12	YES/NO YES-label; component size 200 \| low	23.81 [10.63, 45.09]	21
Qwen3.5-397B-A17B	T12	YES/NO YES-label; component size 250 \| low	14.29 [4.01, 39.94]	14
Qwen3.5-397B-A17B	T12	YES/NO YES-label; component size pooled 100,150,200,250 \| low	26.47 [17.45, 38.01]	68
Qwen3.5-397B-A17B	T12	YES/NO NO-label; component size 100 \| low	100.00 [83.18, 100.00]	19
Qwen3.5-397B-A17B	T12	YES/NO NO-label; component size 150 \| low	100.00 [78.47, 100.00]	14
Qwen3.5-397B-A17B	T12	YES/NO NO-label; component size 200 \| low	100.00 [84.54, 100.00]	21
Qwen3.5-397B-A17B	T12	YES/NO NO-label; component size 250 \| low	100.00 [78.47, 100.00]	14
Qwen3.5-397B-A17B	T12	YES/NO NO-label; component size pooled 100,150,200,250 \| low	100.00 [94.65, 100.00]	68
Qwen3.5-122B-A10B	T12	YES/NO YES-label; component size 100 \| low	31.58 [15.36, 53.99]	19
Qwen3.5-122B-A10B	T12	YES/NO YES-label; component size 150 \| low	14.29 [4.01, 39.94]	14
Qwen3.5-122B-A10B	T12	YES/NO YES-label; component size 200 \| low	14.29 [4.98, 34.64]	21
Qwen3.5-122B-A10B	T12	YES/NO YES-label; component size 250 \| low	7.14 [1.27, 31.47]	14
Qwen3.5-122B-A10B	T12	YES/NO YES-label; component size pooled 100,150,200,250 \| low	17.65 [10.39, 28.36]	68
Qwen3.5-122B-A10B	T12	YES/NO NO-label; component size 100 \| low	100.00 [83.18, 100.00]	19
Qwen3.5-122B-A10B	T12	YES/NO NO-label; component size 150 \| low	100.00 [78.47, 100.00]	14
Qwen3.5-122B-A10B	T12	YES/NO NO-label; component size 200 \| low	100.00 [84.54, 100.00]	21
Qwen3.5-122B-A10B	T12	YES/NO NO-label; component size 250 \| low	100.00 [78.47, 100.00]	14
Qwen3.5-122B-A10B	T12	YES/NO NO-label; component size pooled 100,150,200,250 \| low	100.00 [94.65, 100.00]	68
Qwen3.5-35B-A3B	T12	YES/NO YES-label; component size 100 \| low	21.05 [8.51, 43.33]	19
Qwen3.5-35B-A3B	T12	YES/NO YES-label; component size 150 \| low	21.43 [7.57, 47.59]	14
Qwen3.5-35B-A3B	T12	YES/NO YES-label; component size 200 \| low	14.29 [4.98, 34.64]	21
Qwen3.5-35B-A3B	T12	YES/NO YES-label; component size 250 \| low	7.14 [1.27, 31.47]	14
Qwen3.5-35B-A3B	T12	YES/NO YES-label; component size pooled 100,150,200,250 \| low	16.18 [9.28, 26.69]	68
Qwen3.5-35B-A3B	T12	YES/NO NO-label; component size 100 \| low	100.00 [83.18, 100.00]	19
Qwen3.5-35B-A3B	T12	YES/NO NO-label; component size 150 \| low	92.86 [68.53, 98.73]	14
Qwen3.5-35B-A3B	T12	YES/NO NO-label; component size 200 \| low	100.00 [84.54, 100.00]	21
Qwen3.5-35B-A3B	T12	YES/NO NO-label; component size 250 \| low	100.00 [78.47, 100.00]	14
Qwen3.5-35B-A3B	T12	YES/NO NO-label; component size pooled 100,150,200,250 \| low	98.53 [92.13, 99.74]	68

GraphNIAH v1 · Table 8 · GraphNIAH v1 · Table 9 · GraphNIAH v1 · Table 10 · GraphNIAH v1 · Table 12 · GraphNIAH v1 · Table 24

Context-length experiments

Context-length experiments · paper v1 accuracy (%) with 95% Wilson intervals
Model	Source table	Condition / effort	Accuracy [95% interval]	Evaluated examples N
GPT-5.6-Luna	T25	all three reachability tasks; target context=500k tokens \| low	44.00 [31.16, 57.69]	50
GPT-5.6-Luna	T25	all three reachability tasks; target context=922k tokens \| low	40.00 [27.61, 53.82]	50
Gemini-3.8-Flash	T25	all three reachability tasks; target context=500k tokens \| low	64.00 [50.14, 75.86]	50
Gemini-3.8-Flash	T25	all three reachability tasks; target context=922k tokens \| low	60.00 [46.18, 72.39]	50

GraphNIAH v1 · Table 25

Task and answer-option experiments

Task and answer-option experiments · paper v1 accuracy (%) with 95% Wilson intervals
Model	Source table	Condition / effort	Accuracy [95% interval]	Evaluated examples N
GPT-5.6-Luna	T26	task=YES/NO \| low	66.00 [56.28, 74.54]	100
GPT-5.6-Luna	T26	task=Multiple choice \| low	68.00 [58.34, 76.33]	100
GPT-5.6-Luna	T26	task=Counting \| low	0.00 [0.00, 3.70]	100
GPT-5.6-Luna	T26	all three tasks pooled \| low	44.67 [39.14, 50.32]	300
Gemini-3.8-Flash	T26	task=YES/NO \| low	87.00 [79.02, 92.24]	100
Gemini-3.8-Flash	T26	task=Multiple choice \| low	97.00 [91.55, 98.97]	100
Gemini-3.8-Flash	T26	task=Counting \| low	55.00 [45.24, 64.39]	100
Gemini-3.8-Flash	T26	all three tasks pooled \| low	79.67 [74.75, 83.83]	300
Qwen3.5-35B-A3B	T26	task=YES/NO \| low	64.00 [54.24, 72.73]	100
Qwen3.5-35B-A3B	T26	task=Multiple choice \| low	72.00 [62.51, 79.86]	100
Qwen3.5-35B-A3B	T26	task=Counting \| low	0.00 [0.00, 3.70]	100
Qwen3.5-35B-A3B	T26	all three tasks pooled \| low	45.33 [39.79, 50.99]	300
GPT-5.6-Luna	T13	YES/NO; ground truth=YES-label \| low	32.00 [20.76, 45.81]	50
GPT-5.6-Luna	T13	YES/NO; ground truth=NO-label \| low	100.00 [92.86, 100.00]	50
Gemini-3.8-Flash	T13	YES/NO; ground truth=YES-label \| low	96.00 [86.54, 98.90]	50
Gemini-3.8-Flash	T13	YES/NO; ground truth=NO-label \| low	78.00 [64.76, 87.25]	50
Qwen3.5-35B-A3B	T13	YES/NO; ground truth=YES-label \| low	28.00 [17.47, 41.67]	50
Qwen3.5-35B-A3B	T13	YES/NO; ground truth=NO-label \| low	100.00 [92.86, 100.00]	50
GPT-5.6-Luna	T16	counting; component-size bin=100–149 \| low	0.00 [0.00, 10.43]	33
GPT-5.6-Luna	T16	counting; component-size bin=150–199 \| low	0.00 [0.00, 10.72]	32
GPT-5.6-Luna	T16	counting; component-size bin=200–250 \| low	0.00 [0.00, 9.89]	35
Gemini-3.8-Flash	T16	counting; component-size bin=100–149 \| low	78.79 [62.25, 89.32]	33
Gemini-3.8-Flash	T16	counting; component-size bin=150–199 \| low	59.38 [42.26, 74.48]	32
Gemini-3.8-Flash	T16	counting; component-size bin=200–250 \| low	28.57 [16.33, 45.06]	35
Qwen3.5-35B-A3B	T16	counting; component-size bin=100–149 \| low	0.00 [0.00, 10.43]	33
Qwen3.5-35B-A3B	T16	counting; component-size bin=150–199 \| low	0.00 [0.00, 10.72]	32
Qwen3.5-35B-A3B	T16	counting; component-size bin=200–250 \| low	0.00 [0.00, 9.89]	35
GPT-5.6-Luna	T17	multiple-choice; appended fifth None-of-the-above option (never correct); matched before/after examples \| low	14.00 [8.53, 22.14]	100
Gemini-3.8-Flash	T17	multiple-choice; appended fifth None-of-the-above option (never correct); matched before/after examples \| low	97.00 [91.55, 98.97]	100
Qwen3.5-35B-A3B	T17	multiple-choice; appended fifth None-of-the-above option (never correct); matched before/after examples \| low	15.00 [9.31, 23.28]	100

GraphNIAH v1 · Table 13 · GraphNIAH v1 · Table 16 · GraphNIAH v1 · Table 17 · GraphNIAH v1 · Table 26

Distractor experiments

Distractor experiments · paper v1 accuracy (%) with 95% Wilson intervals
Model	Source table	Condition / effort	Accuracy [95% interval]	Evaluated examples N
GPT-5.6-Luna	T27	all three tasks; distractor=books \| low	41.00 [31.87, 50.80]	100
GPT-5.6-Luna	T27	all three tasks; distractor=code \| low	41.00 [31.87, 50.80]	100
GPT-5.6-Luna	T27	all three tasks; distractor=disjoint-component \| low	41.00 [31.87, 50.80]	100
GPT-5.6-Luna	T27	all three tasks; distractor=essays \| low	46.00 [36.56, 55.74]	100
GPT-5.6-Luna	T27	all three tasks; distractor=wikipedia \| low	50.00 [40.38, 59.62]	100
GPT-5.6-Luna	T27	all three tasks; distractor=all five pooled \| low	43.80 [39.51, 48.18]	500
Gemini-3.8-Flash	T27	all three tasks; distractor=books \| low	83.00 [74.45, 89.11]	100
Gemini-3.8-Flash	T27	all three tasks; distractor=code \| low	72.00 [62.51, 79.86]	100
Gemini-3.8-Flash	T27	all three tasks; distractor=disjoint-component \| low	77.00 [67.85, 84.16]	100
Gemini-3.8-Flash	T27	all three tasks; distractor=essays \| low	88.00 [80.19, 93.00]	100
Gemini-3.8-Flash	T27	all three tasks; distractor=wikipedia \| low	73.00 [63.57, 80.73]	100
Gemini-3.8-Flash	T27	all three tasks; distractor=all five pooled \| low	78.60 [74.79, 81.97]	500
Qwen3.5-35B-A3B	T27	all three tasks; distractor=books \| low	38.00 [29.10, 47.79]	100
Qwen3.5-35B-A3B	T27	all three tasks; distractor=code \| low	42.00 [32.80, 51.79]	100
Qwen3.5-35B-A3B	T27	all three tasks; distractor=disjoint-component \| low	40.00 [30.94, 49.80]	100
Qwen3.5-35B-A3B	T27	all three tasks; distractor=essays \| low	42.00 [32.80, 51.79]	100
Qwen3.5-35B-A3B	T27	all three tasks; distractor=wikipedia \| low	46.00 [36.56, 55.74]	100
Qwen3.5-35B-A3B	T27	all three tasks; distractor=all five pooled \| low	41.60 [37.36, 45.97]	500
GPT-5.6-Luna	T19	YES/NO; distractor=books \| low	56.41 [40.98, 70.70]	39
GPT-5.6-Luna	T19	YES/NO; distractor=code \| low	65.62 [48.31, 79.59]	32
GPT-5.6-Luna	T19	YES/NO; distractor=disjoint-component \| low	59.38 [42.26, 74.48]	32
GPT-5.6-Luna	T19	YES/NO; distractor=essays \| low	67.74 [50.14, 81.43]	31
GPT-5.6-Luna	T19	YES/NO; distractor=wikipedia \| low	56.52 [36.81, 74.37]	23
GPT-5.6-Luna	T19	YES/NO; distractor=all five pooled \| low	61.15 [53.34, 68.42]	157
GPT-5.6-Luna	T19	Multiple choice; distractor=books \| low	63.33 [45.51, 78.13]	30
GPT-5.6-Luna	T19	Multiple choice; distractor=code \| low	71.43 [52.94, 84.75]	28
GPT-5.6-Luna	T19	Multiple choice; distractor=disjoint-component \| low	78.57 [60.46, 89.79]	28
GPT-5.6-Luna	T19	Multiple choice; distractor=essays \| low	69.44 [53.14, 82.00]	36
GPT-5.6-Luna	T19	Multiple choice; distractor=wikipedia \| low	84.09 [70.63, 92.07]	44
GPT-5.6-Luna	T19	Multiple choice; distractor=all five pooled \| low	74.10 [66.94, 80.16]	166
GPT-5.6-Luna	T19	Counting; distractor=books \| low	0.00 [0.00, 11.03]	31
GPT-5.6-Luna	T19	Counting; distractor=code \| low	0.00 [0.00, 8.76]	40
GPT-5.6-Luna	T19	Counting; distractor=disjoint-component \| low	0.00 [0.00, 8.76]	40
GPT-5.6-Luna	T19	Counting; distractor=essays \| low	0.00 [0.00, 10.43]	33
GPT-5.6-Luna	T19	Counting; distractor=wikipedia \| low	0.00 [0.00, 10.43]	33
GPT-5.6-Luna	T19	Counting; distractor=all five pooled \| low	0.00 [0.00, 2.12]	177
Gemini-3.8-Flash	T19	YES/NO; distractor=books \| low	89.74 [76.42, 95.94]	39
Gemini-3.8-Flash	T19	YES/NO; distractor=code \| low	90.62 [75.78, 96.76]	32
Gemini-3.8-Flash	T19	YES/NO; distractor=disjoint-component \| low	90.62 [75.78, 96.76]	32
Gemini-3.8-Flash	T19	YES/NO; distractor=essays \| low	96.77 [83.81, 99.43]	31
Gemini-3.8-Flash	T19	YES/NO; distractor=wikipedia \| low	69.57 [49.13, 84.40]	23
Gemini-3.8-Flash	T19	YES/NO; distractor=all five pooled \| low	88.54 [82.61, 92.62]	157
Gemini-3.8-Flash	T19	Multiple choice; distractor=books \| low	96.67 [83.33, 99.41]	30
Gemini-3.8-Flash	T19	Multiple choice; distractor=code \| low	96.43 [82.29, 99.37]	28
Gemini-3.8-Flash	T19	Multiple choice; distractor=disjoint-component \| low	96.43 [82.29, 99.37]	28
Gemini-3.8-Flash	T19	Multiple choice; distractor=essays \| low	100.00 [90.36, 100.00]	36
Gemini-3.8-Flash	T19	Multiple choice; distractor=wikipedia \| low	97.73 [88.19, 99.60]	44
Gemini-3.8-Flash	T19	Multiple choice; distractor=all five pooled \| low	97.59 [93.97, 99.06]	166
Gemini-3.8-Flash	T19	Counting; distractor=books \| low	61.29 [43.82, 76.27]	31
Gemini-3.8-Flash	T19	Counting; distractor=code \| low	40.00 [26.35, 55.40]	40
Gemini-3.8-Flash	T19	Counting; distractor=disjoint-component \| low	52.50 [37.50, 67.06]	40
Gemini-3.8-Flash	T19	Counting; distractor=essays \| low	66.67 [49.61, 80.25]	33
Gemini-3.8-Flash	T19	Counting; distractor=wikipedia \| low	42.42 [27.24, 59.19]	33
Gemini-3.8-Flash	T19	Counting; distractor=all five pooled \| low	51.98 [44.65, 59.22]	177
Qwen3.5-35B-A3B	T19	YES/NO; distractor=books \| low	56.41 [40.98, 70.70]	39
Qwen3.5-35B-A3B	T19	YES/NO; distractor=code \| low	71.88 [54.63, 84.44]	32
Qwen3.5-35B-A3B	T19	YES/NO; distractor=disjoint-component \| low	59.38 [42.26, 74.48]	32
Qwen3.5-35B-A3B	T19	YES/NO; distractor=essays \| low	67.74 [50.14, 81.43]	31
Qwen3.5-35B-A3B	T19	YES/NO; distractor=wikipedia \| low	56.52 [36.81, 74.37]	23
Qwen3.5-35B-A3B	T19	YES/NO; distractor=all five pooled \| low	62.42 [54.63, 69.61]	157
Qwen3.5-35B-A3B	T19	Multiple choice; distractor=books \| low	53.33 [36.14, 69.77]	30
Qwen3.5-35B-A3B	T19	Multiple choice; distractor=code \| low	67.86 [49.34, 82.07]	28
Qwen3.5-35B-A3B	T19	Multiple choice; distractor=disjoint-component \| low	75.00 [56.64, 87.32]	28
Qwen3.5-35B-A3B	T19	Multiple choice; distractor=essays \| low	58.33 [42.20, 72.86]	36
Qwen3.5-35B-A3B	T19	Multiple choice; distractor=wikipedia \| low	75.00 [60.56, 85.43]	44
Qwen3.5-35B-A3B	T19	Multiple choice; distractor=all five pooled \| low	66.27 [58.78, 73.02]	166
Qwen3.5-35B-A3B	T19	Counting; distractor=books \| low	0.00 [0.00, 11.03]	31
Qwen3.5-35B-A3B	T19	Counting; distractor=code \| low	0.00 [0.00, 8.76]	40
Qwen3.5-35B-A3B	T19	Counting; distractor=disjoint-component \| low	0.00 [0.00, 8.76]	40
Qwen3.5-35B-A3B	T19	Counting; distractor=essays \| low	0.00 [0.00, 10.43]	33
Qwen3.5-35B-A3B	T19	Counting; distractor=wikipedia \| low	0.00 [0.00, 10.43]	33
Qwen3.5-35B-A3B	T19	Counting; distractor=all five pooled \| low	0.00 [0.00, 2.12]	177

GraphNIAH v1 · Table 19 · GraphNIAH v1 · Table 27

Reasoning-effort experiments

Reasoning-effort experiments · paper v1 accuracy (%) with 95% Wilson intervals
Model	Source table	Condition / effort	Accuracy [95% interval]	Evaluated examples N
GPT-5.6-Luna	T28	same 100 examples across efforts; all three reachability tasks \| low	38.00 [29.10, 47.79]	100
GPT-5.6-Luna	T28	same 100 examples across efforts; all three reachability tasks \| medium	39.00 [30.02, 48.80]	100
GPT-5.6-Luna	T28	same 100 examples across efforts; all three reachability tasks \| high	48.00 [38.46, 57.68]	100
Gemini-3.8-Flash	T28	same 100 examples across efforts; all three reachability tasks \| low	71.00 [61.46, 78.99]	100
Gemini-3.8-Flash	T28	same 100 examples across efforts; all three reachability tasks \| medium	75.00 [65.70, 82.45]	100
Gemini-3.8-Flash	T28	same 100 examples across efforts; all three reachability tasks \| high	66.00 [56.28, 74.54]	100
Qwen3.5-35B-A3B	T28	same 100 examples across efforts; all three reachability tasks \| low	40.00 [30.94, 49.80]	100
Qwen3.5-35B-A3B	T28	same 100 examples across efforts; all three reachability tasks \| medium	39.00 [30.02, 48.80]	100
Qwen3.5-35B-A3B	T28	same 100 examples across efforts; all three reachability tasks \| high	40.00 [30.94, 49.80]	100

GraphNIAH v1 · Table 28

Model	Value	Unit	Protocol	Reported date and basis	Source
Gemini-3.1-Pro	60.5	percent	Paper v1; reasoning_effort=low; temperature=1.0; answer extracted from final \answer{} block and exact-matched to graph ground truth; LiteLLM/Gemini; top_p=0.95; no explicit output-token limit; paper reports flex requests with standard-tier capacity fallback; question-depth experiment; all three reachability tasks; question depth pooled k=4,8,16,32; N=400; target context 200000 tokens; other experimental variables sampled independently/uniformly as paper §4.1; concentration alpha=0.1; source Table 23	2026-10-30 · document_publication	GraphNIAH paper v1 · Table23 pooled question-depth accuracy
Gemini-3.6-Flash	54.75	percent	Paper v1; reasoning_effort=low; temperature=1.0; answer extracted from final \answer{} block and exact-matched to graph ground truth; LiteLLM/Gemini; top_p=0.95; no explicit output-token limit; paper reports flex requests with standard-tier capacity fallback; question-depth experiment; all three reachability tasks; question depth pooled k=4,8,16,32; N=400; target context 200000 tokens; other experimental variables sampled independently/uniformly as paper §4.1; concentration alpha=0.1; source Table 23	2026-10-30 · document_publication	GraphNIAH paper v1 · Table23 pooled question-depth accuracy
Gemini-3.8-Flash	79.0	percent	Paper v1; reasoning_effort=low; temperature=1.0; answer extracted from final \answer{} block and exact-matched to graph ground truth; LiteLLM/Gemini; top_p=0.95; no explicit output-token limit; paper reports flex requests with standard-tier capacity fallback; question-depth experiment; all three reachability tasks; question depth pooled k=4,8,16,32; N=400; target context 200000 tokens; other experimental variables sampled independently/uniformly as paper §4.1; concentration alpha=0.1; source Table 23	2026-10-30 · document_publication	GraphNIAH paper v1 · Table23 pooled question-depth accuracy
GPT-5.6-Luna	44.0	percent	Paper v1; reasoning_effort=low; temperature=1.0; answer extracted from final \answer{} block and exact-matched to graph ground truth; LiteLLM/OpenAI; top_p not set while reasoning; no explicit output-token limit; flex requests with standard-tier capacity fallback as reported; question-depth experiment; all three reachability tasks; question depth pooled k=4,8,16,32; N=400; target context 200000 tokens; other experimental variables sampled independently/uniformly as paper §4.1; concentration alpha=0.1; source Table 23	2026-10-30 · document_publication	GraphNIAH paper v1 · Table23 pooled question-depth accuracy
Qwen3.5-122B-A10B	46.75	percent	Paper v1; reasoning_effort=low; temperature=1.0; answer extracted from final \answer{} block and exact-matched to graph ground truth; LiteLLM/OpenRouter, Alibaba first then DeepInfra FP8 fallback; top_p=0.95, top_k=20, min_p=0, presence_penalty=1.5, maximum 40000 output tokens; question-depth experiment; all three reachability tasks; question depth pooled k=4,8,16,32; N=400; target context 200000 tokens; other experimental variables sampled independently/uniformly as paper §4.1; concentration alpha=0.1; source Table 23	2026-10-30 · document_publication	GraphNIAH paper v1 · Table23 pooled question-depth accuracy
Qwen3.5-35B-A3B	45.25	percent	Paper v1; reasoning_effort=low; temperature=1.0; answer extracted from final \answer{} block and exact-matched to graph ground truth; LiteLLM/OpenRouter, Alibaba first then DeepInfra FP8 fallback; top_p=0.95, top_k=20, min_p=0, presence_penalty=1.5, maximum 40000 output tokens; question-depth experiment; all three reachability tasks; question depth pooled k=4,8,16,32; N=400; target context 200000 tokens; other experimental variables sampled independently/uniformly as paper §4.1; concentration alpha=0.1; source Table 23	2026-10-30 · document_publication	GraphNIAH paper v1 · Table23 pooled question-depth accuracy
Qwen3.5-397B-A17B	48.25	percent	Paper v1; reasoning_effort=low; temperature=1.0; answer extracted from final \answer{} block and exact-matched to graph ground truth; LiteLLM/OpenRouter, Alibaba first then DeepInfra FP8 fallback; top_p=0.95, top_k=20, min_p=0, presence_penalty=1.5, maximum 40000 output tokens; question-depth experiment; all three reachability tasks; question depth pooled k=4,8,16,32; N=400; target context 200000 tokens; other experimental variables sampled independently/uniformly as paper §4.1; concentration alpha=0.1; source Table 23	2026-10-30 · document_publication	GraphNIAH paper v1 · Table23 pooled question-depth accuracy
GPT-5.6-Sol	61.67	percent	Paper v1; reasoning_effort=low; temperature=1.0; answer extracted from final \answer{} block and exact-matched to graph ground truth; LiteLLM/OpenAI; top_p not set while reasoning; no explicit output-token limit; flex requests with standard-tier capacity fallback as reported; question-depth experiment; all three reachability tasks; question depth pooled k=8,16,32; N=300; target context 200000 tokens; other experimental variables sampled independently/uniformly as paper §4.1; concentration alpha=0.1; no k=4 run; pool uses k=8,16,32 only; source Table 23	2026-10-30 · document_publication	GraphNIAH paper v1 · Table23 pooled question-depth accuracy

Find citation paths, not isolated facts

A versioned citation-graph collection

Keep every experiment separate

Model settings and incomplete subsets

What this result does not establish

Question-depth experiments

Component-size experiments

Context-length experiments

Task and answer-option experiments

Distractor experiments

Reasoning-effort experiments

Original publication

Evidence available for this benchmark

Start with these resources

Evaluation context and evidence

Reported results 8

Find citation paths, not isolated facts

A versioned citation-graph collection

Keep every experiment separate

Model settings and incomplete subsets

What this result does not establish

Question-depth experiments

Component-size experiments

Context-length experiments

Task and answer-option experiments

Distractor experiments

Reasoning-effort experiments

Original publication

Evidence available for this benchmark

Start with these resources

Evaluation context and evidence

Reported results 8

Explore related benchmarks

Put this benchmark in context

Six new benchmark pages: context, agents and executable tasks

Six Navers Lab / Einsia benchmarks and the work they test

AI benchmarks and leaderboards are different things

How to compare AI benchmark scores without mixing protocols

Four new agent benchmarks: adaptation, safety and video

Continue exploring