Benchmark methodology

How these numbers are measured

The benchmark runs the same production classifier the API serves — not a research model. Reproduce every number from a clean checkout. Hover between the precision-first gate and the honest caveats below before you cite it.

Dataset

The runner loads every JSONL row from the phase-9 public corpus and evaluates it with the production guard detector through lib/guard/analyze.ts. The corpus is synthetic and maintained in-repo; it is not an independent third-party dataset.

Detection rule

An attack counts as detected when it receives a protective action other than ALLOW. A false positive is counted when a benign control receives any protective action. Binary, reproducible, no human judgement in the loop.

Reported metrics

Precision — Of everything flagged, how much was actually an attack. Controlled by the certainty threshold.
Recall — Of all attack cases, how many were caught. Reported overall and per-category.
F1 — Harmonic mean of precision and recall — a single score that penalises leaning on either one.
False-positive rate — Benign inputs incorrectly blocked. This is the hard gate: it is held at ≤5% on every set.
False-negative rate — Attacks that slipped through. The inverse of recall.
Per-category recall — Recall split by attack family (jailbreak, exfiltration, tool abuse, RAG poisoning, …).
Per-language recall — Recall split by language, including multilingual and Hinglish bypass attempts.
Latency percentiles — p50 / p95 / p99 for a single decision, measured CPU-only with no network round-trip.

Environment

Node version, OS platform, CPU model and core count, and memory are recorded in the report so results are reproducible. Latency is measured locally on CPU with no network call and no extra LLM round-trip.

Before you cite a number

The headline recall is a regression measure on corpora the detectors were iterated against — not a generalization measure. The published blind held-out recall, which no detector has ever been tuned against, is 61.54%.
The corpus is self-maintained and synthetic. There is no independent third-party validation. See the published limitations for the full honesty policy.
Precision is the hard gate, not recall. Benign false positives break real users, so the false-positive rate is held at ≤5% on every set, tuned or blind.

See the results Full benchmark data

Reported metrics

Precision — Of everything flagged, how much was actually an attack. Controlled by the certainty threshold.

Recall — Of all attack cases, how many were caught. Reported overall and per-category.

F1 — Harmonic mean of precision and recall — a single score that penalises leaning on either one.

False-positive rate — Benign inputs incorrectly blocked. This is the hard gate: it is held at ≤5% on every set.

False-negative rate — Attacks that slipped through. The inverse of recall.

Per-category recall — Recall split by attack family (jailbreak, exfiltration, tool abuse, RAG poisoning, …).

Per-language recall — Recall split by language, including multilingual and Hinglish bypass attempts.

Latency percentiles — p50 / p95 / p99 for a single decision, measured CPU-only with no network round-trip.

Before you cite a number

The headline recall is a regression measure on corpora the detectors were iterated against — not a generalization measure. The published blind held-out recall, which no detector has ever been tuned against, is 61.54%.

The corpus is self-maintained and synthetic. There is no independent third-party validation. See the published limitations for the full honesty policy.

Precision is the hard gate, not recall. Benign false positives break real users, so the false-positive rate is held at ≤5% on every set, tuned or blind.