ctokx commited on
Commit
88893ff
·
verified ·
1 Parent(s): b0798fa

Add DATASET_CARD.md

Browse files
Files changed (1) hide show
  1. DATASET_CARD.md +177 -0
DATASET_CARD.md ADDED
@@ -0,0 +1,177 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ task_categories:
4
+ - text-classification
5
+ language:
6
+ - en
7
+ tags:
8
+ - cyber-threat-intelligence
9
+ - mitre-attack
10
+ - multi-label-classification
11
+ - defensive-security
12
+ - blue-team
13
+ size_categories:
14
+ - 10K<n<100K
15
+ annotations_creators:
16
+ - expert-generated
17
+ source_datasets:
18
+ - center-for-threat-informed-defense/tram
19
+ pretty_name: TRAM ATT&CK Multi-Label (cleaned, leak-free splits)
20
+ configs:
21
+ - config_name: document
22
+ data_files:
23
+ - split: train
24
+ path: data/build/document_train.jsonl
25
+ - split: dev
26
+ path: data/build/document_dev.jsonl
27
+ - split: test
28
+ path: data/build/document_test.jsonl
29
+ - config_name: random
30
+ data_files:
31
+ - split: train
32
+ path: data/build/random_train.jsonl
33
+ - split: dev
34
+ path: data/build/random_dev.jsonl
35
+ - split: test
36
+ path: data/build/random_test.jsonl
37
+ ---
38
+
39
+ # TRAM ATT&CK Multi-Label (cleaned, with leak-free splits)
40
+
41
+ Sentence-level multi-label mapping of cyber threat intelligence prose to MITRE
42
+ ATT&CK technique IDs. Derived from MITRE CTID's
43
+ [TRAM](https://github.com/center-for-threat-informed-defense/tram) corpus,
44
+ deduplicated and republished with **two split schemes so that leakage can be
45
+ measured rather than assumed**.
46
+
47
+ Everything here is regenerated by `python scripts/01_build_dataset.py`. No row
48
+ was edited by hand.
49
+
50
+ ## Why this exists
51
+
52
+ The upstream corpus contains 19,178 sentences drawn from only **151 source
53
+ reports**. Threat reports repeat phrasing heavily, both within a report and
54
+ across reports covering the same campaign, so the sentence-level random split
55
+ that most work uses puts near-identical prose on both sides of the evaluation.
56
+
57
+ This release ships both splits and expects you to report both:
58
+
59
+ | Config | Construction | Use it for |
60
+ |---|---|---|
61
+ | `document` | No source report appears in more than one split | **Reporting.** The honest number. |
62
+ | `random` | Naive sentence-level shuffle | Comparison against prior work that split this way |
63
+
64
+ ## Construction
65
+
66
+ | Step | Effect |
67
+ |---|---|
68
+ | Raw sentences | 19,178 |
69
+ | Strip scraped `title: … url: …` headers | 7 rows became empty and were dropped |
70
+ | Deduplicate (case/punctuation-insensitive) | 415 duplicate groups, **714 rows removed** |
71
+ | of which appeared in **more than one document** | **370** |
72
+ | Merge labels across duplicates | 12 labels recovered that a naive drop would have lost |
73
+ | Drop techniques in `<3` documents | `T1557.001` only (1 document), 7 label instances |
74
+ | **Final** | **18,437 sentences · 4,024 labelled (21.8%) · 49 techniques · 151 documents** |
75
+
76
+ Deduplication takes the **union** of labels across duplicates. Two copies of one
77
+ sentence annotated `[T1027]` and `[T1027, T1140]` are one sentence annotated
78
+ inconsistently; keeping the union recovers the label instead of discarding it
79
+ with the duplicate row. The surviving row keeps the first document it appeared
80
+ in, so no sentence can span two documents.
81
+
82
+ `T1557.001` is dropped because it appears in exactly one document. It cannot be
83
+ placed in train *and* test, so it is either untrainable or unevaluable. Dropping
84
+ it and saying so beats reporting F1 = 0.00 for it.
85
+
86
+ ## Splits
87
+
88
+ | Config | Split | Sentences | Labelled |
89
+ |---|---|---|---|
90
+ | `document` | train | 11,668 | 2,684 |
91
+ | `document` | dev | 3,134 | 631 |
92
+ | `document` | test | 3,635 | 709 |
93
+ | `random` | train | 12,905 | 2,837 |
94
+ | `random` | dev | 2,765 | 610 |
95
+ | `random` | test | 2,767 | 577 |
96
+
97
+ The `document` split is built by group-aware multi-label stratification in two
98
+ phases. A plain greedy pass fails here: with a 70/15/15 target the train split
99
+ always shows the largest absolute label deficit, so it absorbs every document
100
+ containing a rare technique and the test set ends up with none. Phase 1
101
+ therefore reserves one document per technique for each split, rarest technique
102
+ first; phase 2 packs the remainder using deficit normalised by split size.
103
+
104
+ Consequences, both worth knowing:
105
+
106
+ - **All 49 techniques appear in all three `document` splits.** The naive
107
+ `random` split does not manage this. `T1072` has zero dev examples, so its
108
+ threshold cannot be tuned.
109
+ - Coverage seeding costs some size balance. The `document` split lands at
110
+ roughly 63/17/20 rather than 70/15/15. Larger dev and test sets make the
111
+ estimates *more* stable, so this was accepted rather than corrected.
112
+
113
+ ## Fields
114
+
115
+ ```json
116
+ {
117
+ "sentence": "The dropper base64-encodes its configuration before writing it to disk.",
118
+ "labels": ["T1027"],
119
+ "doc_title": "NotPetya Technical Analysis A Triple Threat ..."
120
+ }
121
+ ```
122
+
123
+ `labels` is empty for 78.2% of rows. These are **real negatives** and should be
124
+ kept. A detector that fires on every sentence of a threat report is useless to
125
+ an analyst.
126
+
127
+ ## Known limitations
128
+
129
+ - **50 techniques, not the full ATT&CK matrix.** The upstream corpus covers the
130
+ 49 retained here. A technique outside that set will never be predicted.
131
+ - **Heavy long tail.** `T1027` has 678 instances; the rarest retained techniques
132
+ have roughly 20. Macro-averaged metrics are the meaningful ones.
133
+ - **Single-sentence context.** Labels were assigned per sentence, so techniques
134
+ only inferable from surrounding paragraphs are under-represented.
135
+ - **151 documents is not many.** Even the document split is one draw from a
136
+ small pool; treat differences of a point or two as noise.
137
+ - **Annotation is not exhaustive.** Some unlabelled sentences do describe
138
+ techniques. Absolute recall is therefore pessimistic.
139
+
140
+ ## Licence and provenance
141
+
142
+ Derived from the TRAM corpus, released by the MITRE Center for Threat-Informed
143
+ Defense under **Apache-2.0**; this derivative keeps Apache-2.0. The underlying
144
+ sentences are fragments of publicly-published vendor threat reports as
145
+ redistributed by MITRE CTID.
146
+
147
+ Technique names come from
148
+ [MITRE ATT&CK STIX data](https://github.com/mitre-attack/attack-stix-data),
149
+ used under the ATT&CK Terms of Use. ATT&CK® is a registered trademark of The
150
+ MITRE Corporation.
151
+
152
+ [AnnoCTR](https://github.com/boschresearch/anno-ctr-lrec-coling-2024) was
153
+ deliberately **not** merged in. It is CC-BY-SA 4.0, and share-alike would
154
+ relicense this entire dataset and restrict downstream reuse. It remains a good
155
+ candidate for a separately-licensed evaluation split.
156
+
157
+ ## Citation
158
+
159
+ ```bibtex
160
+ @misc{tram_attack_multilabel_clean,
161
+ title = {TRAM ATT&CK Multi-Label (cleaned, leak-free splits)},
162
+ author = {Varol Cagdas Tok},
163
+ year = {2026},
164
+ url = {https://huggingface.co/datasets/ctokx/tram-attack-multilabel-clean},
165
+ note = {Derived from MITRE CTID TRAM, Apache-2.0}
166
+ }
167
+ ```
168
+
169
+ Please also cite the upstream corpus:
170
+
171
+ ```bibtex
172
+ @misc{tram,
173
+ title = {TRAM: Threat Report ATT&CK Mapper},
174
+ author = {{MITRE Center for Threat-Informed Defense}},
175
+ howpublished = {\url{https://github.com/center-for-threat-informed-defense/tram}}
176
+ }
177
+ ```