README.md

---
library_name: transformers
license: cc-by-4.0
base_model: roberta-base
pipeline_tag: text-classification
language:
- en
datasets:
- CIRCL/vulnerability-attack-techniques
tags:
- security
- vulnerability
- cve
- mitre-attack
- cti
- multi-label-classification
- generated_from_trainer
model-index:
- name: vulnerability-attack-technique-classification-roberta-base
  results:
  - task:
      type: text-classification
      name: Multi-label MITRE ATT&CK technique classification
    dataset:
      name: CIRCL/vulnerability-attack-techniques
      type: CIRCL/vulnerability-attack-techniques
      split: test
    metrics:
    - type: recall
      name: Recall@5
      value: 0.6440
    - type: recall
      name: Recall@3
      value: 0.5181
    - type: f1
      name: F1 micro
      value: 0.3899
    - type: f1
      name: F1 macro
      value: 0.1910
---

# vulnerability-attack-technique-classification-roberta-base

Suggests [MITRE ATT&CK](https://attack.mitre.org/) (Enterprise) techniques
from a free-text vulnerability description. This is a multi-label classifier
([roberta-base](https://huggingface.co/roberta-base) with a sigmoid head, one
output per technique) fine-tuned on
[CIRCL/vulnerability-attack-techniques](https://huggingface.co/datasets/CIRCL/vulnerability-attack-techniques),
a gold dataset of 1,207 CVEs whose labels come from expert
[MITRE CTID](https://ctid.mitre.org/) mappings. Given a CVE description it
scores 53 parent techniques (e.g. T1190 *Exploit Public-Facing Application*,
T1505 *Server Software Component*), for use as a ranked list of candidate
techniques for analyst review.

The model is trained with [VulnTrain](https://github.com/vulnerability-lookup/VulnTrain)
and runs in production on the public [Vulnerability-Lookup](https://vulnerability.circl.lu)
instance operated by CIRCL, served locally by
[ML-Gateway](https://github.com/vulnerability-lookup/ML-Gateway): every
vulnerability page has an ATT&CK tab with the model's suggestions (example:
[CVE-2021-44077](https://vulnerability.circl.lu/vuln/CVE-2021-44077#attack)).

The methodology, evaluation protocol, and the negative result on
LLM-assisted label expansion are described in the paper
[*Mapping CVEs to MITRE ATT&CK Techniques: A Curated Gold-Set Classifier and
the Limits of LLM-Assisted Label Expansion*](https://arxiv.org/abs/2607.25572)
(arXiv:2607.25572).

DOI: [10.57967/hf/9623](https://doi.org/10.57967/hf/9623)

## How to use

With VulnTrain, against a live CVE:

```bash
vulntrain-infer-attack-classification --cve CVE-2021-44077 \
    --model CIRCL/vulnerability-attack-technique-classification-roberta-base
```

With plain Transformers:

```python
import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

model_id = "CIRCL/vulnerability-attack-technique-classification-roberta-base"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSequenceClassification.from_pretrained(model_id)
model.eval()

description = (
    "Zoho ManageEngine ServiceDesk Plus before 11306, ServiceDesk Plus MSP "
    "before 10530, and SupportCenter Plus before 11014 are vulnerable to "
    "unauthenticated remote code execution."
)
inputs = tokenizer(description, truncation=True, max_length=512, return_tensors="pt")
with torch.no_grad():
    probs = torch.sigmoid(model(**inputs).logits)[0]

for i in probs.argsort(descending=True)[:5]:
    print(f"{model.config.id2label[int(i)]}  {probs[i]:.4f}")
```

Technique IDs map to names via the
[ATT&CK Enterprise matrix](https://attack.mitre.org/techniques/enterprise/);
`sigmoid >= 0.5` is the prediction threshold used in evaluation, but the
model is most useful as a ranker (see the recall@k numbers below).

## Intended uses & limitations

**Intended**: triage assistance — given a vulnerability description, surface
a short ranked list of candidate ATT&CK techniques for a human analyst to
confirm or reject. This is how it is deployed in Vulnerability-Lookup, where
the UI explicitly flags the output as unverified AI-generated suggestions.

**Limitations**:

- The label space is the 53 parent techniques with at least 5 training
  examples; sub-techniques are collapsed to their parent, and techniques
  outside this vocabulary can never be suggested.
- The gold set skews toward exploited-in-the-wild CVEs (CTID's corpus and
  the KEV catalog), so coverage is best for the techniques common there.
- Trained on 972 CVEs — deliberately label-quality-bound rather than
  data-bound (see the paper's gold-size scaling curve: every metric still
  improves monotonically with more curated rows).
- English descriptions only; input is truncated at 512 tokens.
- Scores are not calibrated probabilities.
- Suggestions are **not** verified mappings; treat them as guidance, never
  as authoritative CTI.

## Training and evaluation data

Labels come from the two public expert sources of CVE→ATT&CK mappings, both
following the CTID [*Mapping ATT&CK to CVE for
Impact*](https://ctid.mitre.org/projects/mapping-attck-to-cve-for-impact)
methodology: the CTID `attack_to_cve` mappings and the CTID Mappings
Explorer KEV mappings (ATT&CK 16.1). Technique IDs revoked since publication
are remapped to their successors via the ATT&CK STIX `revoked-by`
relationships. Descriptions are joined from
[CIRCL/vulnerability-scores](https://huggingface.co/datasets/CIRCL/vulnerability-scores)
(Vulnerability-Lookup). Full details in the
[dataset card](https://huggingface.co/datasets/CIRCL/vulnerability-attack-techniques)
and the [VulnTrain documentation](https://github.com/vulnerability-lookup/VulnTrain/blob/main/docs/attack-techniques-dataset.md).

Splits: 972 train / 106 validation / 118 test examples. **Checkpoint
selection uses the validation split; the test split is touched once, for the
numbers reported here.** The paper documents why this matters: selecting the
best checkpoint on the test split inflates and destabilizes small-test-split
metrics enough to produce spurious conclusions.

Notably, this model was **not** trained on LLM-generated labels: the paper's
controlled experiments show that folding in LLM-labeled CVEs at ≈0.39
agreement with the experts yields no reliable ranking improvement and
measurably degrades rare-technique coverage at scale (the comparison
checkpoint is published as
[...-llm-expanded](https://huggingface.co/CIRCL/vulnerability-attack-technique-classification-roberta-base-llm-expanded)).

## Evaluation

Held-out test split, this checkpoint (seed 42):

| Metric | Value |
|--------|-------|
| Recall@5 | 0.6440 |
| Recall@3 | 0.5181 |
| F1 micro | 0.3899 |
| F1 macro | 0.1910 |
| Precision micro | 0.2740 |
| Recall micro | 0.6756 |

Recall@5 = 0.64 means that on average 64% of an unseen CVE's expert-assigned
techniques appear in the model's top five suggestions — roughly double the
zero-shot embedding-similarity baseline reported in the paper. Across five
seeds under the identical protocol, the numbers of record are recall@5
0.673 ± 0.019, recall@3 0.536 ± 0.032, micro-F1 0.410 ± 0.006, macro-F1
0.177 ± 0.014. The complete trainer logs are published in the
[paper repository](https://github.com/vulnerability-lookup/cve-attack-mapping-paper/tree/master/trainer-logs).

## Training procedure

Binary cross-entropy over 53 sigmoid outputs, with per-label `pos_weight`
balancing (capped at 20) to keep rare techniques trainable. Trained with
`vulntrain-train-attack-classification` (VulnTrain).

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 1e-05
- train_batch_size: 32
- eval_batch_size: 32
- seed: 42
- optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
- lr_scheduler_type: linear
- num_epochs: 40
- max_length: 512
- loss: BCEWithLogitsLoss, balanced pos_weight (min 2.447, max 20.0)
- checkpoint selection: best macro-F1 on the validation split

### Training results

| Training Loss | Epoch | Step | Validation Loss | F1 Micro | F1 Macro | Precision Micro | Recall Micro | Recall At 3 | Recall At 5 |
|:-------------:|:-----:|:----:|:---------------:|:--------:|:--------:|:---------------:|:------------:|:-----------:|:-----------:|
| 0.9311        | 1.0   | 31   | 0.8413          | 0.1833   | 0.0401   | 0.1258          | 0.3376       | 0.1798      | 0.2611      |
| 0.8366        | 2.0   | 62   | 0.7876          | 0.1452   | 0.0234   | 0.1065          | 0.2278       | 0.1047      | 0.2086      |
| 0.8116        | 3.0   | 93   | 0.7717          | 0.2054   | 0.0518   | 0.1641          | 0.2743       | 0.1938      | 0.3234      |
| 0.7941        | 4.0   | 124  | 0.7576          | 0.3204   | 0.0804   | 0.2382          | 0.4895       | 0.3387      | 0.5009      |
| 0.7743        | 5.0   | 155  | 0.7435          | 0.3044   | 0.0841   | 0.2119          | 0.5401       | 0.3313      | 0.4696      |
| 0.7645        | 6.0   | 186  | 0.7290          | 0.3208   | 0.0906   | 0.2174          | 0.6118       | 0.3969      | 0.5391      |
| 0.7472        | 7.0   | 217  | 0.7163          | 0.3551   | 0.1130   | 0.2571          | 0.5738       | 0.4068      | 0.5741      |
| 0.7219        | 8.0   | 248  | 0.7056          | 0.3224   | 0.1079   | 0.2173          | 0.6245       | 0.4079      | 0.5521      |
| 0.7195        | 9.0   | 279  | 0.6933          | 0.3576   | 0.1495   | 0.2449          | 0.6624       | 0.4252      | 0.5663      |
| 0.6835        | 10.0  | 310  | 0.6845          | 0.3705   | 0.1665   | 0.2579          | 0.6582       | 0.4708      | 0.6090      |
| 0.6539        | 11.0  | 341  | 0.6768          | 0.4063   | 0.1810   | 0.2947          | 0.6540       | 0.5227      | 0.6318      |
| 0.6484        | 12.0  | 372  | 0.6725          | 0.3632   | 0.1734   | 0.2520          | 0.6498       | 0.4449      | 0.6200      |
| 0.6249        | 13.0  | 403  | 0.6664          | 0.3974   | 0.1782   | 0.2862          | 0.6498       | 0.5034      | 0.6396      |
| 0.6109        | 14.0  | 434  | 0.6585          | 0.3801   | 0.1721   | 0.2724          | 0.6287       | 0.4834      | 0.6491      |
| 0.6004        | 15.0  | 465  | 0.6539          | 0.3872   | 0.1678   | 0.2781          | 0.6371       | 0.4752      | 0.6347      |
| 0.5896        | 16.0  | 496  | 0.6502          | 0.4049   | 0.1777   | 0.2996          | 0.6245       | 0.4768      | 0.6397      |
| 0.5667        | 17.0  | 527  | 0.6478          | 0.3866   | 0.1682   | 0.2737          | 0.6582       | 0.4941      | 0.6472      |
| 0.5661        | 18.0  | 558  | 0.6425          | 0.4108   | 0.1910   | 0.3022          | 0.6414       | 0.5128      | 0.6667      |
| 0.5501        | 19.0  | 589  | 0.6394          | 0.3880   | 0.1861   | 0.2758          | 0.6540       | 0.4822      | 0.6561      |
| 0.5461        | 20.0  | 620  | 0.6377          | 0.4097   | 0.1804   | 0.3010          | 0.6414       | 0.5069      | 0.6687      |
| 0.5351        | 21.0  | 651  | 0.6338          | 0.4028   | 0.1715   | 0.3002          | 0.6118       | 0.4987      | 0.6624      |
| 0.5215        | 22.0  | 682  | 0.6351          | 0.4146   | 0.1964   | 0.3054          | 0.6456       | 0.4943      | 0.6875      |
| 0.5155        | 23.0  | 713  | 0.6315          | 0.4056   | 0.1737   | 0.3023          | 0.6160       | 0.4994      | 0.6553      |
| 0.5063        | 24.0  | 744  | 0.6269          | 0.4286   | 0.1879   | 0.3208          | 0.6456       | 0.5195      | 0.6923      |
| 0.5061        | 25.0  | 775  | 0.6264          | 0.4178   | 0.1869   | 0.3069          | 0.6540       | 0.5246      | 0.6656      |
| 0.4996        | 26.0  | 806  | 0.6301          | 0.4073   | 0.1792   | 0.3053          | 0.6118       | 0.5274      | 0.6958      |
| 0.4950        | 27.0  | 837  | 0.6225          | 0.4133   | 0.1771   | 0.3079          | 0.6287       | 0.5376      | 0.6593      |
| 0.4928        | 28.0  | 868  | 0.6228          | 0.4173   | 0.1768   | 0.3166          | 0.6118       | 0.5187      | 0.7048      |
| 0.4819        | 29.0  | 899  | 0.6242          | 0.4263   | 0.1871   | 0.3225          | 0.6287       | 0.5494      | 0.6970      |
| 0.4766        | 30.0  | 930  | 0.6194          | 0.4166   | 0.1862   | 0.3094          | 0.6371       | 0.5226      | 0.6871      |
| 0.4694        | 31.0  | 961  | 0.6213          | 0.42     | 0.1914   | 0.3175          | 0.6203       | 0.5399      | 0.6918      |
| 0.4781        | 32.0  | 992  | 0.6209          | 0.4292   | 0.1893   | 0.3281          | 0.6203       | 0.5439      | 0.7060      |
| 0.4636        | 33.0  | 1023 | 0.6218          | 0.4347   | 0.1962   | 0.3276          | 0.6456       | 0.525       | 0.6797      |
| 0.4641        | 34.0  | 1054 | 0.6216          | 0.4314   | 0.1887   | 0.3261          | 0.6371       | 0.5415      | 0.6797      |
| 0.4592        | 35.0  | 1085 | 0.6206          | 0.4313   | 0.1916   | 0.3282          | 0.6287       | 0.5466      | 0.6858      |
| 0.4526        | 36.0  | 1116 | 0.6208          | 0.4357   | 0.1926   | 0.3333          | 0.6287       | 0.5447      | 0.6863      |
| 0.4643        | 37.0  | 1147 | 0.6191          | 0.4218   | 0.1899   | 0.3196          | 0.6203       | 0.5392      | 0.6863      |
| 0.4501        | 38.0  | 1178 | 0.6191          | 0.4242   | 0.1885   | 0.3224          | 0.6203       | 0.5368      | 0.6929      |
| 0.4570        | 39.0  | 1209 | 0.6206          | 0.4350   | 0.1932   | 0.3326          | 0.6287       | 0.5281      | 0.6910      |
| 0.4436        | 40.0  | 1240 | 0.6199          | 0.4325   | 0.1916   | 0.3296          | 0.6287       | 0.5329      | 0.6882      |

### Framework versions

- Transformers 5.13.0
- Pytorch 2.12.1+cu130
- Datasets 4.8.5
- Tokenizers 0.22.2

## Related artifacts

| Artifact | Location | DOI |
|----------|----------|-----|
| Gold dataset (1,207 CVEs, CTID-curated labels) | [CIRCL/vulnerability-attack-techniques](https://huggingface.co/datasets/CIRCL/vulnerability-attack-techniques) | [10.57967/hf/9621](https://doi.org/10.57967/hf/9621) |
| LLM expansion dataset (negative result) | [CIRCL/vulnerability-attack-techniques-llm-scaling](https://huggingface.co/datasets/CIRCL/vulnerability-attack-techniques-llm-scaling) | [10.57967/hf/9622](https://doi.org/10.57967/hf/9622) |
| LLM-expanded comparison model | [CIRCL/vulnerability-attack-technique-classification-roberta-base-llm-expanded](https://huggingface.co/CIRCL/vulnerability-attack-technique-classification-roberta-base-llm-expanded) | [10.57967/hf/9624](https://doi.org/10.57967/hf/9624) |
| Code | [vulnerability-lookup/VulnTrain](https://github.com/vulnerability-lookup/VulnTrain) | — |
| Paper | [arXiv:2607.25572](https://arxiv.org/abs/2607.25572) | — |
| Paper LaTeX source + trainer logs | [vulnerability-lookup/cve-attack-mapping-paper](https://github.com/vulnerability-lookup/cve-attack-mapping-paper) | — |

## Citation

```bibtex
@misc{bonhomme2026mappingcvesmitreattck,
      title={Mapping CVEs to MITRE ATT&CK Techniques: A Curated Gold-Set Classifier and the Limits of LLM-Assisted Label Expansion},
      author={Cédric Bonhomme and Alexandre Dulaunoy},
      year={2026},
      eprint={2607.25572},
      archivePrefix={arXiv},
      primaryClass={cs.CR},
      url={https://arxiv.org/abs/2607.25572},
}
```

## Acknowledgements

Developed at [CIRCL](https://www.circl.lu) in the context of the
[AIPITCH](https://www.science.nask.pl/en/research-areas/projects/12456)
project, co-funded by the European Union.