File size: 4,231 Bytes
26b6b80
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
---
library_name: peft
pipeline_tag: text-generation
tags:
  - lora
  - qlora
  - peft
  - vulnerability-detection
  - security
  - cwe
  - llvm-ir
  - code
language:
  - en
base_model:
  - Qwen/Qwen2.5-Coder-7B-Instruct
license: apache-2.0
---

# Vulnerability Detector — IR vs source adapters (Phase 3B)

> **Status: valid research artifacts.** Both adapters are functional, but their
> purpose is to reproduce the one genuine positive finding of the thesis
> (Section 16): training on LLVM IR detects worse yet classifies the CWE far
> better than training on the matched C source. Use them to reproduce that
> dissociation, not as a general detector.

From the MSc thesis *Software Vulnerability Detection with Advanced Machine
Learning Techniques* (Claudio Bendini, University of Parma, 2025–2026, advisor
Prof.ssa Eleonora Iotti). Both adapters fine-tune the same base model,
Qwen2.5-Coder-7B-Instruct, on the same SARD Juliet programs, one arm on the C
source and the other on the `clang -O0 -emit-llvm` intermediate representation.

- **Code, data pipeline, full results:** https://github.com/Phazertron/Vulnerability_Detector
- **Canonical record:** [`docs/EXPERIMENT_LOG.md`](https://github.com/Phazertron/Vulnerability_Detector/blob/main/docs/EXPERIMENT_LOG.md), Section 16.
- **Companion repos:** [Phase 2 SFT](https://huggingface.co/phazertron/vulnerability-detector-sft) (working detectors) and [DPO](https://huggingface.co/phazertron/vulnerability-detector-dpo-negative) (a negative result).

## The finding

On the parity split (246 records both arms could handle):

| Metric | `source` arm | `ir` arm | Gap |
|---|---|---|---|
| Binary F1 (detection) | 0.5069 | 0.3116 | -0.1954 |
| CWE accuracy given detection | 0.1455 | **0.7742** | **+0.6287** |

Restricted to the 11 records both arms detected correctly, CWE accuracy is
1/11 for source against 8/11 for IR (McNemar exact p = 0.039; 95% CI on the gap
[+0.18, +1.00], excludes zero). IR is **worse at deciding whether code is
vulnerable** and **much better at naming which weakness it is**, and the
defensible claim is the +0.18 lower bound, not the point estimate. The effect
spreads across 13 CWEs rather than concentrating on memory-safety ones.

## The adapters

| Subfolder | Trained on | Role |
|---|---|---|
| `ir` | SARD Juliet, LLVM IR | the IR arm (strong CWE attribution) |
| `source` | SARD Juliet, C source | the matched baseline for comparison |

Both are for reproducing Section 16. Neither is a strong real-world detector:
they are trained on synthetic Juliet programs and do not transfer to real-world
code (Section 14).

## How to load

Both use base `Qwen/Qwen2.5-Coder-7B-Instruct`. Rank 32, alpha 64, dropout 0.05,
all attention and MLP projections, 4-bit base (QLoRA), 2048-token sequence
budget. PEFT 0.13.2, TRL 1.5.0, Transformers 4.57.6.

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel

base_id = "Qwen/Qwen2.5-Coder-7B-Instruct"
repo    = "phazertron/vulnerability-detector-ir-source"
sub     = "ir"   # or "source"

bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
                         bnb_4bit_compute_dtype=torch.bfloat16)
tok = AutoTokenizer.from_pretrained(repo, subfolder=sub)
base = AutoModelForCausalLM.from_pretrained(base_id, quantization_config=bnb, device_map="auto")
model = PeftModel.from_pretrained(base, repo, subfolder=sub)
```

The `source` arm is prompted with the C function; the `ir` arm with the
function's LLVM IR (see the pipeline in the GitHub repo). Expected output is one
line: `VULNERABLE: CWE-787` or `CLEAN`.

## Training data

[SARD Juliet](https://samate.nist.gov/SARD/test-suites/112) (compilable
synthetic C). The adapters contain learned weights, not dataset content.

## Licensing

Both adapters derive from Qwen2.5-Coder-7B-Instruct and are released under
[Apache 2.0](https://huggingface.co/Qwen/Qwen2.5-Coder-7B-Instruct/blob/main/LICENSE).

## Citation

```bibtex
@mastersthesis{bendini2026vulndetect,
  author = {Claudio Bendini},
  title  = {Software Vulnerability Detection with Advanced Machine Learning Techniques},
  school = {University of Parma},
  year   = {2026}
}
```