Frozen version 1.0, published November 2, 2026; four scored gateway configurations.
What does the gateway miss?
Argus-Bench tests whether an input-inspection gateway flags prompt-injection attempts before they reach an agent. Its separate benign controls reveal whether the same policy also blocks ordinary requests. It measures input detection, not whether a downstream agent ultimately follows an attack.
204 attacks and 60 controls
The frozen v1.0 contains 264 English, single-turn cases across seven attack families. All but one were authored for this benchmark; A52 adapts a public persona attack. Labels received author self-review, without an independently human-judged validation set.
Detection must be read with false positives
Both flag and block labels count as detected under the stated flag-and-audit policy. The primary series is detection over all 204 attacks. It excludes benign controls from that denominator; control false-positive rates and attack-family rates are separate measurements below.
Results by attack family
Read the original numerators and denominators alongside percentages. A perfect result on one family is not an overall score.
| Category | Cases | Argus | LLM Guard | Rebuff heuristic | LLM judge |
|---|---|---|---|---|---|
| direct_override | 33 | 3/33 (9.1%) | 33/33 (100.0%) | 2/33 (6.1%) | 15/33 (45.5%) |
| role_override | 28 | 5/28 (17.9%) | 15/28 (53.6%) | 0/28 (0.0%) | 18/28 (64.3%) |
| system_prompt_probe | 25 | 3/25 (12.0%) | 23/25 (92.0%) | 0/25 (0.0%) | 13/25 (52.0%) |
| encoded_payload | 29 | 3/29 (10.3%) | 25/29 (86.2%) | 1/29 (3.4%) | 16/29 (55.2%) |
| gradual_escalation | 39 | 0/39 (0.0%) | 24/39 (61.5%) | 1/39 (2.6%) | 21/39 (53.8%) |
| indirect_contextual | 30 | 0/30 (0.0%) | 7/30 (23.3%) | 0/30 (0.0%) | 16/30 (53.3%) |
| tool_misuse | 20 | 1/20 (5.0%) | 4/20 (20.0%) | 0/20 (0.0%) | 7/20 (35.0%) |
| All attacks / 全部攻击 | 204 | 15/204 (7.35%) | 131/204 (64.22%) | 4/204 (1.96%) | 106/204 (51.96%) |
Percentages are copied as printed, without recomputation or rounding changes.
What normal requests get flagged?
Low detection can coexist with low false positives. The benign controls deliberately resemble the attack surface, including security discussions and questions about encoding.
| System | False positives / controls | FP rate | Case IDs |
|---|---|---|---|
| Argus | 7/60 | 11.67% | C04, C06, C07, C10, C16, C28, C43 |
| LLM Guard | 10/60 | 16.67% | C02, C14, C19, C23, C24, C30, C42, C44, C49, C54 |
| Rebuff heuristic | 0/60 | 0.00% | — |
| LLM judge | 19/60 | 31.67% | C03, C04, C07, C08, C09, C11, C13, C17, C19, C23, C24, C25, C27, C37, C46, C48, C49, C53, C57 |
These are false-positive rates, not attack-detection scores. Lower is better for this separate metric.
The configuration is part of the result
Argus uses five regular-expression patterns. LLM Guard 0.3.16 uses protectai/deberta-v3-base-prompt-injection-v2 at threshold 0.92 with full-text matching. Rebuff 0.1.1 uses only its offline heuristic layer at threshold 0.75; its hosted model and vector-database layers were not tested. The judge is Qwen2.5-0.5B-Instruct, Q4_K_M through llama.cpp, temperature 0 and a fixed binary classification prompt.
Why NeMo has no score
The author implemented NeMo Guardrails 0.24 with a local Qwen2.5-0.5B self_check_input rail but did not complete a full-corpus run. Smoke-test observations and a runtime estimate are not a benchmark score; no zero or inferred result is added here.
What this release can support
This is a small, author-constructed diagnostic dataset for gateway input filtering. It does not establish resilience to multi-turn attacks, multilingual traffic or downstream tool execution. Rebuff here is a component configuration, not the complete product. The original release includes the dataset, README and result summary; the companion paper is named but no verified publication URL was found in this deposit.
Original dataset README · Original frozen result tables and system configurations · Argus-Bench versioned original deposit
Release and review dates
The versioned Zenodo deposit was published on November 2, 2026 and reviewed for this page on November 3. That publication date timestamps the reported results; the exact experiment execution dates are not stated.