Public benchmark

SoterAI AI Security Benchmark

This benchmark is maintained by SoterAI. It is not an independent third-party benchmark unless explicitly stated. Results may not represent all real-world attacks.

100.00%

Latest recall

0.00%

False-positive rate

3,200

Dataset rows

26.98 ms

p95 latency

What This Benchmark Measures

The runner loads public JSONL cases and evaluates each prompt with the production SoterAI guard detector. A protective action other than ALLOW counts as a detection for attack rows and as a false positive for benign rows.

Dataset Summary

Total cases: 3,200

Attack cases: 2,200

Benign controls: 1,000

Source type: synthetic

Dataset path: benchmarks\soterai-public-benchmark

Metrics

Precision: 100.00%

Recall: 100.00%

F1: 1.0000

False-negative rate: 0.00%

Thresholds passed: YES

Attack Categories

Category	Detected	Recall
data-exfiltration	300/300	100.00%
hinglish-multilingual	200/200	100.00%
jailbreak	250/250	100.00%
mcp-risk	100/100	100.00%
prompt-injection	250/250	100.00%
rag-poisoning	200/200	100.00%
secret-pii	300/300	100.00%
system-prompt-leak	300/300	100.00%
tool-abuse	100/100	100.00%
unicode-obfuscation	200/200	100.00%

Benign Control Categories

normal-developer-promptssecurity-educationbusiness-writingcoding-helppublic-info

Latest Results

Generated 2026-07-15T06:32:00.556Z. Confusion matrix: TP 2200, FP 0, TN 1000, FN 0.

Latency

p50 10.74 ms, p95 26.98 ms, p99 54.58 ms.

Hardware / Environment

Node v22.16.0; Windows_NT 10.0.26300 x64; Intel(R) Core(TM) i5-8350U CPU @ 1.70GHz; 8 CPU threads; 15.81 GB memory.

Methodology

Reproduce with node scripts/phase-9-run-public-benchmark.js. The script exports JSON, Markdown, and CSV artifacts and fails if recall, FPR, or latency thresholds are not met.

Comparison Methodology

No competitor superiority claim is made here. Like-for-like competitor comparison requires the same dataset, same labels, same threshold policy, and documented runtime environment.

Limitations

This benchmark is maintained by SoterAI.
It is not an independent third-party benchmark unless explicitly stated.
Results may not represent all real-world attacks.
The public dataset is synthetic and does not include production traffic.
External penetration testing and independent validation are tracked separately.
No AI security tool can guarantee complete protection against every possible attack.

What This Benchmark Does Not Prove

External penetration testing and independent validation are tracked separately. No AI security tool can guarantee complete protection against every possible attack. This benchmark does not prove SOC2 compliance, production GA readiness, marketplace approval, or best-in-world status.

External Validation Status

No independent third-party benchmark or external pentest report is claimed by this page. External validation remains evidence required before stronger claims are approved.

Reproducibility

Dataset files are under benchmarks/soterai-public-benchmark. Results are under benchmarks/results.

Download Results

JSON CSV Methodology

Changelog

2026-07-15: Phase 9 public benchmark dataset, runner, and results page added.