Volpidash's picture
Publish security benchmark protocol, measured coverage and available results
8a2e047 verified
|
Raw History Blame Contribute Delete
3.94 kB

Dataset coverage audit

Counts describe retained data, not demonstrated model capabilities.

Measure Training Held out
Examples 3396 264
Distinct CVE IDs mentioned 1190 93
iOS 27 / 27.x advisory examples 326 0
CISA vendor labels 158 29
CISA vendor/product pairs 264 36

CVE totals count unique identifiers mentioned, not all publicly known vulnerabilities or independently verified facts. Vendor counts use the source's labels.

Gaps and interpretation

  • No iOS 27 advisory examples are in the current held-out split; iOS 27 generalization remains unmeasured.
  • Apple examples primarily teach advisory interpretation, not kernel/WebKit source-code auditing.
  • Original MCQ answer positions are heavily skewed toward A; the separate benchmark reorders eligible choices to balance answer letters.
  • Exact group/CVE and lexical overlap checks cannot rule out semantic overlap or base-model pretraining contamination.

Split checks: 0 shared groups / 0 shared CVE IDs. These checks do not establish absence of all data leakage.

iOS releases represented

The version names the release containing a fix, not an affected-version range. These are advisory tasks, not source-code audit tasks.

iOS release Training examples Held-out examples
27.2 163 0
27 163 0
26.7.2 134 0
26.7.1 1 0
26.7 95 0
26.6.1 35 0
26.6 97 0
26.5.2 38 0
26.5 67 0
26.4 44 0
26.3 43 0
26.2 0 41
18.7.11 3 0
18.7.10 131 0
18.7.9 49 0
18.7.7 26 0
18.7.5 29 0
18.7.3 2 0
16.7.16 0 1
16.7.15 1 0
15.8.7 4 0

Most frequent Apple component labels

Top 15 by training-example count. All component counts are in coverage.json.

Advisory component Training examples Held-out examples
WebKit 201 11
Kernel 150 1
ImageIO 62 0
Model I/O 35 0
SceneKit 30 0
CoreMedia 22 0
AVEVideoEncoder 21 0
CoreAudio 18 0
Wi-Fi 16 0
Security 15 1
WebRTC 15 0
Audio 11 0
mDNSResponder 11 0
Accounts 10 0
FontParser 10 0

Other vendor coverage

Top 15 CISA vendor labels by training count.

Vendor label Training examples Held-out examples
Microsoft 67 10
Cisco 27 0
Fortinet 16 0
Linux 15 1
Google 14 1
Ivanti 13 1
Citrix 11 0
Adobe 10 0
Synacor 9 0
SonicWall 9 1
Oracle 7 1
D-Link 6 0
Apache 6 0
Broadcom 5 1
Palo Alto Networks 5 0

Topic keywords

Non-exclusive keyword matches in prompt plus answer. A row can match several topics; these are heuristic tags, not expert annotations.

Keyword topic Training matches Held-out matches
iOS / Apple platform 1178 47
Kernel / operating systems 948 69
Memory safety / binary analysis 664 25
Web / API security 681 69
Network security 502 48
Identity / authentication 463 52
Cryptography 203 11
Cloud / containers 575 55
Malware / incident analysis 276 26

Original multiple-choice answer positions

The original training data is unchanged. Eligible benchmark choices are reordered to balance answer letters independently in each evaluation suite.

Correct letter Training Held out
A 1167 113
B 502 47
C 123 17
D 25 3

Reproducibility

Machine-readable counts and corpus SHA-256 hashes. Audit generated 2026-11-09T13:40:21.918181+00:00.