Text Classification
Transformers
Safetensors
English
modernbert
cyber-threat-intelligence
mitre-attack
multi-label-classification
defensive-security
blue-team
threat-intelligence
text-embeddings-inference
Instructions to use ctokx/cti-attack-mapper-modernbert with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use ctokx/cti-attack-mapper-modernbert with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-classification", model="ctokx/cti-attack-mapper-modernbert")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForSequenceClassification tokenizer = AutoTokenizer.from_pretrained("ctokx/cti-attack-mapper-modernbert") model = AutoModelForSequenceClassification.from_pretrained("ctokx/cti-attack-mapper-modernbert", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Add DATASET_CARD.md
Browse files- DATASET_CARD.md +177 -0
DATASET_CARD.md
ADDED
|
@@ -0,0 +1,177 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: apache-2.0
|
| 3 |
+
task_categories:
|
| 4 |
+
- text-classification
|
| 5 |
+
language:
|
| 6 |
+
- en
|
| 7 |
+
tags:
|
| 8 |
+
- cyber-threat-intelligence
|
| 9 |
+
- mitre-attack
|
| 10 |
+
- multi-label-classification
|
| 11 |
+
- defensive-security
|
| 12 |
+
- blue-team
|
| 13 |
+
size_categories:
|
| 14 |
+
- 10K<n<100K
|
| 15 |
+
annotations_creators:
|
| 16 |
+
- expert-generated
|
| 17 |
+
source_datasets:
|
| 18 |
+
- center-for-threat-informed-defense/tram
|
| 19 |
+
pretty_name: TRAM ATT&CK Multi-Label (cleaned, leak-free splits)
|
| 20 |
+
configs:
|
| 21 |
+
- config_name: document
|
| 22 |
+
data_files:
|
| 23 |
+
- split: train
|
| 24 |
+
path: data/build/document_train.jsonl
|
| 25 |
+
- split: dev
|
| 26 |
+
path: data/build/document_dev.jsonl
|
| 27 |
+
- split: test
|
| 28 |
+
path: data/build/document_test.jsonl
|
| 29 |
+
- config_name: random
|
| 30 |
+
data_files:
|
| 31 |
+
- split: train
|
| 32 |
+
path: data/build/random_train.jsonl
|
| 33 |
+
- split: dev
|
| 34 |
+
path: data/build/random_dev.jsonl
|
| 35 |
+
- split: test
|
| 36 |
+
path: data/build/random_test.jsonl
|
| 37 |
+
---
|
| 38 |
+
|
| 39 |
+
# TRAM ATT&CK Multi-Label (cleaned, with leak-free splits)
|
| 40 |
+
|
| 41 |
+
Sentence-level multi-label mapping of cyber threat intelligence prose to MITRE
|
| 42 |
+
ATT&CK technique IDs. Derived from MITRE CTID's
|
| 43 |
+
[TRAM](https://github.com/center-for-threat-informed-defense/tram) corpus,
|
| 44 |
+
deduplicated and republished with **two split schemes so that leakage can be
|
| 45 |
+
measured rather than assumed**.
|
| 46 |
+
|
| 47 |
+
Everything here is regenerated by `python scripts/01_build_dataset.py`. No row
|
| 48 |
+
was edited by hand.
|
| 49 |
+
|
| 50 |
+
## Why this exists
|
| 51 |
+
|
| 52 |
+
The upstream corpus contains 19,178 sentences drawn from only **151 source
|
| 53 |
+
reports**. Threat reports repeat phrasing heavily, both within a report and
|
| 54 |
+
across reports covering the same campaign, so the sentence-level random split
|
| 55 |
+
that most work uses puts near-identical prose on both sides of the evaluation.
|
| 56 |
+
|
| 57 |
+
This release ships both splits and expects you to report both:
|
| 58 |
+
|
| 59 |
+
| Config | Construction | Use it for |
|
| 60 |
+
|---|---|---|
|
| 61 |
+
| `document` | No source report appears in more than one split | **Reporting.** The honest number. |
|
| 62 |
+
| `random` | Naive sentence-level shuffle | Comparison against prior work that split this way |
|
| 63 |
+
|
| 64 |
+
## Construction
|
| 65 |
+
|
| 66 |
+
| Step | Effect |
|
| 67 |
+
|---|---|
|
| 68 |
+
| Raw sentences | 19,178 |
|
| 69 |
+
| Strip scraped `title: … url: …` headers | 7 rows became empty and were dropped |
|
| 70 |
+
| Deduplicate (case/punctuation-insensitive) | 415 duplicate groups, **714 rows removed** |
|
| 71 |
+
| of which appeared in **more than one document** | **370** |
|
| 72 |
+
| Merge labels across duplicates | 12 labels recovered that a naive drop would have lost |
|
| 73 |
+
| Drop techniques in `<3` documents | `T1557.001` only (1 document), 7 label instances |
|
| 74 |
+
| **Final** | **18,437 sentences · 4,024 labelled (21.8%) · 49 techniques · 151 documents** |
|
| 75 |
+
|
| 76 |
+
Deduplication takes the **union** of labels across duplicates. Two copies of one
|
| 77 |
+
sentence annotated `[T1027]` and `[T1027, T1140]` are one sentence annotated
|
| 78 |
+
inconsistently; keeping the union recovers the label instead of discarding it
|
| 79 |
+
with the duplicate row. The surviving row keeps the first document it appeared
|
| 80 |
+
in, so no sentence can span two documents.
|
| 81 |
+
|
| 82 |
+
`T1557.001` is dropped because it appears in exactly one document. It cannot be
|
| 83 |
+
placed in train *and* test, so it is either untrainable or unevaluable. Dropping
|
| 84 |
+
it and saying so beats reporting F1 = 0.00 for it.
|
| 85 |
+
|
| 86 |
+
## Splits
|
| 87 |
+
|
| 88 |
+
| Config | Split | Sentences | Labelled |
|
| 89 |
+
|---|---|---|---|
|
| 90 |
+
| `document` | train | 11,668 | 2,684 |
|
| 91 |
+
| `document` | dev | 3,134 | 631 |
|
| 92 |
+
| `document` | test | 3,635 | 709 |
|
| 93 |
+
| `random` | train | 12,905 | 2,837 |
|
| 94 |
+
| `random` | dev | 2,765 | 610 |
|
| 95 |
+
| `random` | test | 2,767 | 577 |
|
| 96 |
+
|
| 97 |
+
The `document` split is built by group-aware multi-label stratification in two
|
| 98 |
+
phases. A plain greedy pass fails here: with a 70/15/15 target the train split
|
| 99 |
+
always shows the largest absolute label deficit, so it absorbs every document
|
| 100 |
+
containing a rare technique and the test set ends up with none. Phase 1
|
| 101 |
+
therefore reserves one document per technique for each split, rarest technique
|
| 102 |
+
first; phase 2 packs the remainder using deficit normalised by split size.
|
| 103 |
+
|
| 104 |
+
Consequences, both worth knowing:
|
| 105 |
+
|
| 106 |
+
- **All 49 techniques appear in all three `document` splits.** The naive
|
| 107 |
+
`random` split does not manage this. `T1072` has zero dev examples, so its
|
| 108 |
+
threshold cannot be tuned.
|
| 109 |
+
- Coverage seeding costs some size balance. The `document` split lands at
|
| 110 |
+
roughly 63/17/20 rather than 70/15/15. Larger dev and test sets make the
|
| 111 |
+
estimates *more* stable, so this was accepted rather than corrected.
|
| 112 |
+
|
| 113 |
+
## Fields
|
| 114 |
+
|
| 115 |
+
```json
|
| 116 |
+
{
|
| 117 |
+
"sentence": "The dropper base64-encodes its configuration before writing it to disk.",
|
| 118 |
+
"labels": ["T1027"],
|
| 119 |
+
"doc_title": "NotPetya Technical Analysis A Triple Threat ..."
|
| 120 |
+
}
|
| 121 |
+
```
|
| 122 |
+
|
| 123 |
+
`labels` is empty for 78.2% of rows. These are **real negatives** and should be
|
| 124 |
+
kept. A detector that fires on every sentence of a threat report is useless to
|
| 125 |
+
an analyst.
|
| 126 |
+
|
| 127 |
+
## Known limitations
|
| 128 |
+
|
| 129 |
+
- **50 techniques, not the full ATT&CK matrix.** The upstream corpus covers the
|
| 130 |
+
49 retained here. A technique outside that set will never be predicted.
|
| 131 |
+
- **Heavy long tail.** `T1027` has 678 instances; the rarest retained techniques
|
| 132 |
+
have roughly 20. Macro-averaged metrics are the meaningful ones.
|
| 133 |
+
- **Single-sentence context.** Labels were assigned per sentence, so techniques
|
| 134 |
+
only inferable from surrounding paragraphs are under-represented.
|
| 135 |
+
- **151 documents is not many.** Even the document split is one draw from a
|
| 136 |
+
small pool; treat differences of a point or two as noise.
|
| 137 |
+
- **Annotation is not exhaustive.** Some unlabelled sentences do describe
|
| 138 |
+
techniques. Absolute recall is therefore pessimistic.
|
| 139 |
+
|
| 140 |
+
## Licence and provenance
|
| 141 |
+
|
| 142 |
+
Derived from the TRAM corpus, released by the MITRE Center for Threat-Informed
|
| 143 |
+
Defense under **Apache-2.0**; this derivative keeps Apache-2.0. The underlying
|
| 144 |
+
sentences are fragments of publicly-published vendor threat reports as
|
| 145 |
+
redistributed by MITRE CTID.
|
| 146 |
+
|
| 147 |
+
Technique names come from
|
| 148 |
+
[MITRE ATT&CK STIX data](https://github.com/mitre-attack/attack-stix-data),
|
| 149 |
+
used under the ATT&CK Terms of Use. ATT&CK® is a registered trademark of The
|
| 150 |
+
MITRE Corporation.
|
| 151 |
+
|
| 152 |
+
[AnnoCTR](https://github.com/boschresearch/anno-ctr-lrec-coling-2024) was
|
| 153 |
+
deliberately **not** merged in. It is CC-BY-SA 4.0, and share-alike would
|
| 154 |
+
relicense this entire dataset and restrict downstream reuse. It remains a good
|
| 155 |
+
candidate for a separately-licensed evaluation split.
|
| 156 |
+
|
| 157 |
+
## Citation
|
| 158 |
+
|
| 159 |
+
```bibtex
|
| 160 |
+
@misc{tram_attack_multilabel_clean,
|
| 161 |
+
title = {TRAM ATT&CK Multi-Label (cleaned, leak-free splits)},
|
| 162 |
+
author = {Varol Cagdas Tok},
|
| 163 |
+
year = {2026},
|
| 164 |
+
url = {https://huggingface.co/datasets/ctokx/tram-attack-multilabel-clean},
|
| 165 |
+
note = {Derived from MITRE CTID TRAM, Apache-2.0}
|
| 166 |
+
}
|
| 167 |
+
```
|
| 168 |
+
|
| 169 |
+
Please also cite the upstream corpus:
|
| 170 |
+
|
| 171 |
+
```bibtex
|
| 172 |
+
@misc{tram,
|
| 173 |
+
title = {TRAM: Threat Report ATT&CK Mapper},
|
| 174 |
+
author = {{MITRE Center for Threat-Informed Defense}},
|
| 175 |
+
howpublished = {\url{https://github.com/center-for-threat-informed-defense/tram}}
|
| 176 |
+
}
|
| 177 |
+
```
|