Paper v1 Table 2: primary results are the five all-test-case weighted Overall scores; all task and reliability values remain below.
From remembering facts to acting appropriately
Five tasks test single-source recall, multi-source recall, temporal updates, personal-pattern inference and responses to requests conflicting with personal constraints. This is a response benchmark, not live tool execution.
P-ContextBench paper v1 · P-ContextBench author code and sample pipeline
Synthetic personal histories
The paper uses 100 U.S.-seeded personas, 10,088 hidden facts, 18,261 messenger/calendar fragments and 4,810 verified tests. Logs contain 2.5 filler fragments per evidence fragment and no decoys. Models see complete relevant logs without retrieval.
Mean credit is not repeatable full correctness
A separate LLM judge checks evidence and task-specific requirements. Overall accuracy averages awarded credit; Pass³ requires full credit in all three runs. These metrics must remain separate. Source dates are paper-publication dates, not run timestamps.
Overall accuracy · every task and model
All five systems and every T1–T5 column from Table 2 are retained exactly. Avg. is the authors’ test-count-weighted average across all 4,810 cases.
| Model | T1 | T2 | T3 | T4 | T5 | Avg. · all test cases |
|---|---|---|---|---|---|---|
| GPT-5.4-mini | 78.98 | 66.22 | 49.05 | 54.83 | 46.05 | 56.25 |
| Gemini-3-Flash | 89.09 | 74.74 | 50.68 | 54.69 | 54.63 | 61.61 |
| Claude Haiku 4.5 | 84.5 | 72.7 | 50.0 | 59.3 | 50.8 | 60.75 |
| Qwen3.5-35B-A3B | 75.87 | 57.99 | 39.17 | 47.45 | 35.72 | 48.1 |
| Claude Sonnet 4.6 | 94.78 | 86.03 | 72.90 | 74.53 | 64.16 | 75.52 |
T1 single-source; T2 multi-source; T3 temporal state; T4 personal-pattern inference; T5 constraint-aware responses. Compare the same metric and task; the three tables are different metrics.
Worst-run accuracy · every task and model
All five systems and every T1–T5 column from Table 2 are retained exactly. Avg. is the authors’ test-count-weighted average across all 4,810 cases.
| Model | T1 | T2 | T3 | T4 | T5 | Avg. · all test cases |
|---|---|---|---|---|---|---|
| GPT-5.4-mini | 73.09 | 60.59 | 44.49 | 51.70 | 41.12 | 51.66 |
| Gemini-3-Flash | 85.60 | 70.55 | 46.46 | 52.05 | 51.49 | 58.27 |
| Claude Haiku 4.5 | 84.2 | 72.1 | 49.6 | 58.5 | 50.3 | 60.19 |
| Qwen3.5-35B-A3B | 69.63 | 51.96 | 34.13 | 44.21 | 29.67 | 42.96 |
| Claude Sonnet 4.6 | 94.19 | 85.10 | 72.73 | 73.33 | 63.08 | 74.62 |
T1 single-source; T2 multi-source; T3 temporal state; T4 personal-pattern inference; T5 constraint-aware responses. Compare the same metric and task; the three tables are different metrics.
Pass³ accuracy · every task and model
All five systems and every T1–T5 column from Table 2 are retained exactly. Avg. is the authors’ test-count-weighted average across all 4,810 cases.
| Model | T1 | T2 | T3 | T4 | T5 | Avg. · all test cases |
|---|---|---|---|---|---|---|
| GPT-5.4-mini | 50.89 | 22.22 | 2.63 | 3.37 | 0.85 | 11.64 |
| Gemini-3-Flash | 72.57 | 39.81 | 5.72 | 5.60 | 2.98 | 18.75 |
| Claude Haiku 4.5 | 59.3 | 28.0 | 5.1 | 5.5 | 0.6 | 14.46 |
| Qwen3.5-35B-A3B | 43.23 | 10.84 | 0.60 | 0.07 | 0.09 | 7.57 |
| Claude Sonnet 4.6 | 84.85 | 59.09 | 30.81 | 32.35 | 4.76 | 34.65 |
T1 single-source; T2 multi-source; T3 temporal state; T4 personal-pattern inference; T5 constraint-aware responses. Compare the same metric and task; the three tables are different metrics.
Version and reporting limits
Table 2 gives Sonnet 75.52 overall and 34.65 Pass³, while narrative states 75.49 and 34.58; the table is retained without reconciliation. Synthetic U.S.-seeded logs and model judging do not establish reliability for real users.
P-ContextBench paper v1 · P-ContextBench author code and sample pipeline