Free IDE extension for risk-free vibe coding — keep secrets out of AI.
Adversarial Benchmark
Measured with the exact production classifier over a disclosed corpus of 1,218 cases (108 attacks, 1,110 benign). We publish the metric that actually matters — recall at a fixed false-positive rate — not a “100%” score. Reproduce it with npm run benchmark:honest.
How to read this
A guard is only useful if it catches attacks without flooding real users with false blocks. Recall@1%FPR answers exactly that: “if we tune the guard so at most 1 in 100 benign messages is flagged, what fraction of real attacks do we still catch?” It is the metric industry benchmarks (Lakera PINT, Meta PromptGuard-2) report — and it is far more honest than a headline accuracy number.
Recall @ 1% FPR
Attacks caught within a 1% false-positive budget
ROC-AUC
Threshold-independent separability
Precision
FPR 0.8% on 1,110 benign
p50 Latency
Analyzer CPU time, no network
98.1%
Attacks blocked, reviewed, rewritten or redacted
63.9%
Strictest — fully stopped or escalated
Static single-turn benchmarks miss the most realistic attack shape: the slow escalation where each message is individually mild but the session as a whole builds toward a bypass. We evaluate the session-level defense separately across 10 conversations (including benign sessions that reuse “as we discussed / go deeper” phrasing, to measure false escalation honestly).
Multi-turn recall
Escalating attack sessions caught
Benign escalation rate
Benign sessions wrongly escalated
Mean turns to catch
How fast the session defense fires
Weakest categories first — we show where the guard is strong and where it has room to improve, because an honest benchmark is how detection actually gets better.
2 attack prompts
1 attack prompt
2 attack prompts
4 attack prompts
4 attack prompts
2 attack prompts
2 attack prompts
11 attack prompts
7 attack prompts
1 attack prompt
2 attack prompts
1 attack prompt
1 attack prompt
1 attack prompt
2 attack prompts
1 attack prompt
1 attack prompt
1 attack prompt
1 attack prompt
2 attack prompts
1 attack prompt
2 attack prompts
1 attack prompt
2 attack prompts
2 attack prompts
2 attack prompts
2 attack prompts
2 attack prompts
2 attack prompts
2 attack prompts
2 attack prompts
2 attack prompts
2 attack prompts
4 attack prompts
2 attack prompts
1 attack prompt
4 attack prompts
7 attack prompts
7 attack prompts
4 attack prompts
3 attack prompts
3 attack prompts
Every case is labeled with its source so you can audit exactly what these numbers were measured against.
guardRedTeamBenchmark
70
68 attack · 2 benign
phase5Benchmark
48
40 attack · 8 benign
expandedSafeInputs
1,100
0 attack · 1,100 benign
Per-input analyzer CPU time (deterministic engine, no model, no network). The one-off cold-start outlier (~2022ms max) is JIT/module warm-up, not steady-state.
p50 (Median)
4.6ms
p95
12.1ms
p99
42.3ms
Separate from detection accuracy: an end-to-end regression suite asserting each security service rejects its abuse cases — agent firewall bypass, passport forgery, delegation abuse, egress exfiltration, evidence tampering. It exits non-zero if any scenario regresses.
Scenarios passing
Across 21 services
Services covered
Guard, agents, identity, evidence, SIEM
Failing scenarios
Run June 29, 2026
Command: npx tsx tests/comprehensive-adversarial-test-battery.ts. This is a service-hardening suite, not a detection-accuracy score — those are the honest metrics above.
The exact production classifier (analyzeText) scores every case. We sweep its risk score to find the threshold that keeps benign false positives at or below 1%, then report attack recall there (Recall@1%FPR), plus ROC-AUC, per-category recall, a multi-turn Crescendo evaluation, and latency. Reproduce with npm run benchmark:honest.
Try the interactive playground, then protect both sides of your model call.
Full raw results available at /api/benchmarks (JSON).
