OASB Skills Security Benchmark
A corpus and methodology for evaluating AI agent skill scanners. 9 attack categories, sourced from DVAA scenarios, ARIA research findings and expert-reviewed payloads.
Dataset: v2.0 | Comparative scanner scores withdrawn 2026-08-09 (see below) | Paper comparison: Holzbauer et al. (arXiv:2603.16572)
The comparative accuracy figures on this page have been withdrawn
This page previously reported an F1 score and a false-positive rate for the HackMyAgent pipeline, and a scanner leaderboard. Those figures are withdrawn as of 2026-08-09 because the false-positive rate was an artifact of how the corpus was labeled, not a measurement of scanner behaviour.
3,704 of the 3,881 benign samples were labeled benign by the rule verdict='warning' AND overall_score >= 70 as reported by HackMyAgent itself. Any artifact HackMyAgent would have flagged was therefore excluded from the benign class by construction, so a near-zero false-positive rate was guaranteed before any scan ran. A scanner cannot be graded against labels it produced.
We are re-measuring on corpora we do not own and did not label, and will republish only figures that a third party can reproduce without our cooperation. Until then no comparative claim is made here. The corpus, the taxonomy and the methodology remain available and useful; only the scores are withdrawn.
Recall was withdrawn too. It should not have been, and it is back
The withdrawal above rests on a specific fact: the benign class was labeled by the scanner under test. That fact is fatal to any metric that reads the benign class, which is false-positive rate, precision, F1 and flag rate. It says nothing about recall, which reads only the malicious class. The first version of this notice removed the recall figures along with the rest, justified by “a scanner cannot be graded against labels it produced”. That sentence is true of the withdrawn metrics and false of recall, so withdrawing recall on that ground was itself inaccurate.
The published run already excluded the 225 registry samples labeled malicious by the same rule. What it scored was 270 attack fixtures, and every one of them is ours: 89 from ARIA research, 91 from DVAA, and 90 that are HackMyAgent’s own test payloads.
| Malicious samples scored | Detected | Recall |
|---|---|---|
| ARIA research findings | 74 / 89 | 83.1% |
| DVAA scenarios | 74 / 91 | 81.3% |
| HackMyAgent’s own test payloads | 75 / 90 | 83.3% |
| All authored fixtures (published figure) | 223 / 270 | 82.6% |
| Excluding HackMyAgent’s own payloads | 148 / 180 | 82.2% |
| Including the 225 self-labeled samples | 234 / 495 | 47.3% |
Two things that table is here to let you check rather than take on trust. A third of the denominator is the scanner’s own test payloads, and they are not inflating the result: 83.3% on its own payloads against 81.3% on DVAA. And the exclusion of the 225 self-labeled samples moves the number a great deal. We exclude them because they carry no attack content, being registry entries HackMyAgent happened to block, and scoring against them would be the same circularity pointing the other way. If you disagree with that exclusion, the number you want is 47.3%.
This measures detection against a fixed set of fixtures we wrote. It is not a measure of detection in the wild, and it is not comparable to a figure another scanner reports on another corpus. Per-category results are published as counts rather than percentages, in BENCHMARK-RESULTS.md section 1a: each category is 30 samples, so a one-decimal percentage on a single category claims more precision than the sample size supports.
Verdict: attack findings, not posture
A sample is flagged malicious on a high/critical attack finding. Posture findings that fire on benign and malicious artifacts alike are excluded from the verdict (the scanner still surfaces them to users): missing prompt/governance defenses, and "allowedTools": ["*"] wildcard tool access, which 2,900+ benign registry MCP servers also declare -- a least-privilege posture issue, not malice. That is the same treatment already applied to missing-governance checks. The previously shown 82.1% F1 / 1.26% FPR (a skill-routing artifact that bypassed the MCP analyzers) and the older 89.2% figure are withdrawn, as is the 82.9% F1 that replaced them.
Why this benchmark exists
Holzbauer et al. evaluated 9 scanners across 238,180 skills from 3 marketplaces. Flag rates ranged from 3.8% to 41.9%, but only 33 out of 27,111 skills (0.12%) were flagged by all scanners. No scanner reported precision, recall, or F1 because no ground-truth labeled dataset existed.
OASB was built to provide that ground truth across 9 attack categories, sourced from DVAA scenarios, ARIA research findings, expert-reviewed payloads and real registry data. The attack samples are expert-authored and hold up. The benign class does not: most of it was labeled by HackMyAgent’s own verdict, which is why the scores derived from it are withdrawn above. Rebuilding the benign class from independently labeled sources is the current work, and until it lands OASB should be read as a corpus and a taxonomy rather than as a scoreboard.
DVAA Ground-Truth Validation
The Damn Vulnerable AI Agent scenarios provide a second ground-truth set of intentionally vulnerable agents, each with a known attack type.
Across the full DVAA scenario repo (86 scenarios, real attack files), the structural pipeline detects 29.1% under the same verdict (a high or critical attack finding). On the config-structural DVAA samples carried in the corpus (91 samples) it reaches 81.3%. The gap is the honest picture: the structural analyzers catch config-encoded attacks (self-escalation, control-bypass and credential-harvest directives) but miss most behavioral and natural-language attacks, which depend on the semantic layer. Lead with the 29.1% full-repo figure when characterizing DVAA detection.
Industry Comparison
Scanner flag rates from Holzbauer et al. (arXiv:2603.16572), 238,180 skills across 3 marketplaces. These scanners report flag rates only; no precision/recall is available (no ground truth).
| Scanner | Platform | Flag Rate | Precision | Recall | Flag Rate |
|---|---|---|---|---|---|
| Socket | Skills.sh | 3.8% | -- | -- | |
| Snyk | Skills.sh | 7.7% | -- | -- | |
| agent-trust-hub | Skills.sh | 13.8% | -- | -- | |
| Cisco Skill Scanner | Skills.sh | 14.0% | -- | -- | |
| Cisco Skill Scanner | ClawHub | 16.7% | -- | -- | |
| GPT 5.3-based LLM | Skills.sh | 27.3% | -- | -- | |
| VirusTotal | ClawHub | 36.2% | -- | -- | |
| GPT 5.3-based LLM | ClawHub | 38.8% | -- | -- | |
| OpenClaw Scanner | ClawHub | 41.9% | -- | -- |
Paper scanners tested on 238K real marketplace skills (no labels). Flag rate only — precision and recall require ground truth. Our own rows have been removed from this table because the corpus they were scored on labeled its benign class using the scanner under test.
Methodology
Dataset Composition
- 270 malicious samples (30 per category)
- 3,881 benign samples from real registries
- 94 edge cases (security tools, defensive configs)
- Sources: DVAA scenarios, ARIA research, HMA payloads, expert review, registry scans
- 225 registry metadata-flagged stubs excluded (no malicious content)
Scoring
- Binary detection: malicious/benign verdict per sample
- Category assignment: 9 attack categories for malicious verdicts
- Metrics: pooled F1, precision, recall over all samples (micro-average)
- FPR: false positives / (false positives + true negatives)
- Edge case samples excluded from scoring
Submit Your Scanner
The benchmark is open. Any scanner can submit results for evaluation. The methodology is the authority, not a gating decision.
Submissions expire after 90 days. Scanners must resubmit against each new dataset version to maintain their rating.
POST https://api.oa2a.org/api/v1/benchmark/submit
Content-Type: application/json
{
"scannerId": "your-scanner-id",
"scannerName": "Your Scanner",
"scannerVersion": "1.0.0",
"datasetVersion": "v2.0",
"results": [
{ "sampleId": "m001", "verdict": "malicious", "category": "supply_chain" },
...
]
}References
- Holzbauer et al., "Malicious Or Not: Adding Repository Context to Agent Skill Classification," arXiv:2603.16572, March 2026
- OASB benchmark code and dataset: github.com/opena2a-org/oasb
- DVAA scenarios: github.com/opena2a-org/damn-vulnerable-ai-agent