Volpidash's picture
Publish security benchmark protocol, measured coverage and available results
8a2e047 verified
|
Raw History Blame Contribute Delete
3.94 kB
# Dataset coverage audit
Counts describe retained data, not demonstrated model capabilities.
| Measure | Training | Held out |
| --- | --- | --- |
| Examples | 3396 | 264 |
| Distinct CVE IDs mentioned | 1190 | 93 |
| iOS 27 / 27.x advisory examples | 326 | 0 |
| CISA vendor labels | 158 | 29 |
| CISA vendor/product pairs | 264 | 36 |
CVE totals count unique identifiers mentioned, not all publicly known vulnerabilities or independently verified facts. Vendor counts use the source's labels.
## Gaps and interpretation
- No iOS 27 advisory examples are in the current held-out split; iOS 27 generalization remains unmeasured.
- Apple examples primarily teach advisory interpretation, not kernel/WebKit source-code auditing.
- Original MCQ answer positions are heavily skewed toward A; the separate benchmark reorders eligible choices to balance answer letters.
- Exact group/CVE and lexical overlap checks cannot rule out semantic overlap or base-model pretraining contamination.
Split checks: **0 shared groups / 0 shared CVE IDs**. These checks do not establish absence of all data leakage.
## iOS releases represented
The version names the **release containing a fix**, not an affected-version range. These are advisory tasks, not source-code audit tasks.
| iOS release | Training examples | Held-out examples |
| --- | --- | --- |
| 27.2 | 163 | 0 |
| 27 | 163 | 0 |
| 26.7.2 | 134 | 0 |
| 26.7.1 | 1 | 0 |
| 26.7 | 95 | 0 |
| 26.6.1 | 35 | 0 |
| 26.6 | 97 | 0 |
| 26.5.2 | 38 | 0 |
| 26.5 | 67 | 0 |
| 26.4 | 44 | 0 |
| 26.3 | 43 | 0 |
| 26.2 | 0 | 41 |
| 18.7.11 | 3 | 0 |
| 18.7.10 | 131 | 0 |
| 18.7.9 | 49 | 0 |
| 18.7.7 | 26 | 0 |
| 18.7.5 | 29 | 0 |
| 18.7.3 | 2 | 0 |
| 16.7.16 | 0 | 1 |
| 16.7.15 | 1 | 0 |
| 15.8.7 | 4 | 0 |
## Most frequent Apple component labels
Top 15 by training-example count. All component counts are in [coverage.json](./coverage.json).
| Advisory component | Training examples | Held-out examples |
| --- | --- | --- |
| WebKit | 201 | 11 |
| Kernel | 150 | 1 |
| ImageIO | 62 | 0 |
| Model I/O | 35 | 0 |
| SceneKit | 30 | 0 |
| CoreMedia | 22 | 0 |
| AVEVideoEncoder | 21 | 0 |
| CoreAudio | 18 | 0 |
| Wi-Fi | 16 | 0 |
| Security | 15 | 1 |
| WebRTC | 15 | 0 |
| Audio | 11 | 0 |
| mDNSResponder | 11 | 0 |
| Accounts | 10 | 0 |
| FontParser | 10 | 0 |
## Other vendor coverage
Top 15 CISA vendor labels by training count.
| Vendor label | Training examples | Held-out examples |
| --- | --- | --- |
| Microsoft | 67 | 10 |
| Cisco | 27 | 0 |
| Fortinet | 16 | 0 |
| Linux | 15 | 1 |
| Google | 14 | 1 |
| Ivanti | 13 | 1 |
| Citrix | 11 | 0 |
| Adobe | 10 | 0 |
| Synacor | 9 | 0 |
| SonicWall | 9 | 1 |
| Oracle | 7 | 1 |
| D-Link | 6 | 0 |
| Apache | 6 | 0 |
| Broadcom | 5 | 1 |
| Palo Alto Networks | 5 | 0 |
## Topic keywords
Non-exclusive keyword matches in prompt plus answer. A row can match several topics; these are heuristic tags, not expert annotations.
| Keyword topic | Training matches | Held-out matches |
| --- | --- | --- |
| iOS / Apple platform | 1178 | 47 |
| Kernel / operating systems | 948 | 69 |
| Memory safety / binary analysis | 664 | 25 |
| Web / API security | 681 | 69 |
| Network security | 502 | 48 |
| Identity / authentication | 463 | 52 |
| Cryptography | 203 | 11 |
| Cloud / containers | 575 | 55 |
| Malware / incident analysis | 276 | 26 |
## Original multiple-choice answer positions
The original training data is unchanged. Eligible benchmark choices are reordered to balance answer letters independently in each evaluation suite.
| Correct letter | Training | Held out |
| --- | --- | --- |
| A | 1167 | 113 |
| B | 502 | 47 |
| C | 123 | 17 |
| D | 25 | 3 |
## Reproducibility
[Machine-readable counts and corpus SHA-256 hashes](./coverage.json). Audit generated 2026-11-09T13:40:21.918181+00:00.