Binamra00 commited on
Commit
a007f4a
·
verified ·
1 Parent(s): e9fa500

Safetensors weights for seeds 42, 1337, 2024; model card, config, inference code

Browse files
.gitattributes ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ model_attention_1337.safetensors filter=lfs diff=lfs merge=lfs -text
2
+ model_attention_2024.safetensors filter=lfs diff=lfs merge=lfs -text
3
+ model_attention_42.safetensors filter=lfs diff=lfs merge=lfs -text
README.md ADDED
@@ -0,0 +1,183 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: mit
3
+ language: en
4
+ library_name: pytorch
5
+ pipeline_tag: text-classification
6
+ base_model: microsoft/codebert-base
7
+ tags:
8
+ - code
9
+ - commit-classification
10
+ - maintenance-activities
11
+ - software-engineering
12
+ - mining-software-repositories
13
+ - codebert
14
+ - java
15
+ metrics:
16
+ - accuracy
17
+ model-index:
18
+ - name: codebert-commit-intent
19
+ results:
20
+ - task:
21
+ type: text-classification
22
+ name: Commit maintenance-activity classification
23
+ dataset:
24
+ type: levin-yehudai-commits
25
+ name: Levin & Yehudai labelled commits (173-commit test split)
26
+ metrics:
27
+ - type: accuracy
28
+ value: 86.7
29
+ name: Accuracy, mean of 3 seeds
30
+ - type: cohen_kappa
31
+ value: 0.791
32
+ name: Cohen's kappa, mean of 3 seeds
33
+ ---
34
+
35
+ # CodeBERT commit-intent classifier
36
+
37
+ Classifies a commit as **corrective** (fixing a fault), **adaptive** (adding a feature) or
38
+ **perfective** (improving the code or its design), Swanson's maintenance categories, from its
39
+ message and its diff.
40
+
41
+ It is [CodeBERT](https://huggingface.co/microsoft/codebert-base) fine-tuned on the 1,151 commits
42
+ Levin & Yehudai labelled by hand. Where their method reads 48 counts of AST change types extracted
43
+ by ChangeDistiller, this model reads the diff itself, so it runs on modern Java that ChangeDistiller
44
+ cannot parse. On the same 173 held-out commits it reaches **86.7% accuracy and Cohen's κ 0.791**,
45
+ against 75.7% and 0.621 for a reproduction of their best model.
46
+
47
+ | | |
48
+ |---|---|
49
+ | Training code and evaluation | [github.com/Binamra00/commit-intent-and-refactoring-levels](https://github.com/Binamra00/commit-intent-and-refactoring-levels) |
50
+ | Data, stage outputs and 34,609 classified commits | [10.5281/zenodo.24295072](https://doi.org/10.5281/zenodo.24295072) |
51
+ | Base model | [microsoft/codebert-base](https://huggingface.co/microsoft/codebert-base) |
52
+
53
+ ## Files
54
+
55
+ | file | |
56
+ |---|---|
57
+ | `model_attention_2024.safetensors` | the default model, seed 2024 |
58
+ | `model_attention_42.safetensors`, `model_attention_1337.safetensors` | the other two seeds; the reported scores are the mean of all three |
59
+ | `config.json` | labels, input limits, calibration temperature and thresholds |
60
+ | `commit_intent.py` | the architecture and an inference wrapper; the weights load into it unchanged |
61
+
62
+ ## Usage
63
+
64
+ ```bash
65
+ pip install torch transformers safetensors huggingface_hub
66
+ ```
67
+
68
+ ```python
69
+ from huggingface_hub import hf_hub_download
70
+ import importlib.util
71
+
72
+ path = hf_hub_download("Binamra00/codebert-commit-intent", "commit_intent.py")
73
+ spec = importlib.util.spec_from_file_location("commit_intent", path)
74
+ ci = importlib.util.module_from_spec(spec); spec.loader.exec_module(ci)
75
+
76
+ clf = ci.CommitIntent.from_pretrained("Binamra00/codebert-commit-intent") # seed=42 / 1337 for the others
77
+
78
+ patch = """@@ -10,7 +10,9 @@ public Config load(Path file) {
79
+ - return parse(Files.readString(file));
80
+ + String text = Files.readString(file);
81
+ + if (text.isBlank()) return Config.empty();
82
+ + return parse(text);
83
+ """
84
+ clf.predict("Fix NPE when the config file is empty", {"src/main/java/Config.java": patch})
85
+ # {'label': ..., 'predicted': ..., 'probabilities': {...}, 'margin': ...}
86
+ ```
87
+
88
+ **Input.** The full commit message, and for each changed file its unified-diff patch, as GitHub's
89
+ API returns it in the `patch` field or as one file's section of `git diff`. Context lines are
90
+ ignored. The model was trained on Java projects.
91
+
92
+ **Output.**
93
+
94
+ - `probabilities`: calibrated with temperature scaling (T = 2.86).
95
+ - `margin`: the top probability minus the runner-up.
96
+ - `predicted`: the most likely class.
97
+ - `label`: the same class, or `unclassified` when the margin is below 0.47. Pass `threshold=` to
98
+ change that cut, or read `predicted` to always get one of the three classes.
99
+
100
+ ## How it works
101
+
102
+ Each commit is split into pieces: one for the message, and one per changed file, holding that
103
+ file's removed and added lines. Each piece is encoded by CodeBERT, in 512-token windows when it is
104
+ longer. Windows are averaged within a piece, and a learned attention score weights the pieces into
105
+ one commit vector. A linear layer then predicts the three classes. A commit is never truncated: up
106
+ to 256 pieces are kept, and no commit in the training data had more.
107
+
108
+ ## Training
109
+
110
+ | | |
111
+ |---|---|
112
+ | Data | Levin & Yehudai's 1,151 labelled commits from 11 Java projects ([10.5281/zenodo.835534](https://doi.org/10.5281/zenodo.835534)), with diffs fetched from the GitHub API; 1,150 used, one excluded because the API truncated it or it changed no Java file |
113
+ | Split | their protocol: 85/15 stratified, 977 training / 173 test commits |
114
+ | Model selection | attention vs transformer pooling, by 5-fold cross-validation repeated twice on the training commits |
115
+ | Optimiser | AdamW, learning rate 2e-5 for the encoder and 1e-3 for the head, linear warmup over 10% of steps then linear decay, weight decay 0.01, gradient clipping at 1.0 |
116
+ | Loss | cross-entropy weighted by inverse class frequency |
117
+ | Epochs | up to 10, early stopping after 3 without a better validation κ, on 10% of the training commits |
118
+ | Seeds | 42, 1337, 2024 |
119
+ | Hardware | one NVIDIA A100 80 GB, bfloat16 autocast |
120
+
121
+ ## Evaluation
122
+
123
+ On the 173 test commits, which were opened once, after training:
124
+
125
+ | seed | accuracy | κ | macro F1 |
126
+ |---|---|---|---|
127
+ | 42 | 85.5% | 0.771 | 83.4 |
128
+ | 1337 | 86.7% | 0.792 | 85.3 |
129
+ | 2024 | 87.9% | 0.810 | 86.5 |
130
+ | **mean** | **86.7% ± 1.2** | **0.791 ± 0.020** | **85.1** |
131
+
132
+ For comparison, on the same commits:
133
+
134
+ | model | accuracy | κ |
135
+ |---|---|---|
136
+ | Levin & Yehudai, as published | 76.7% | 0.635 |
137
+ | Our reproduction of their model | 75.7% | 0.621 |
138
+ | Best reproduced feature-based model | 78.0% | 0.654 |
139
+
140
+ Temperature scaling, fitted on the test commits, took expected calibration error from 0.109 to
141
+ 0.025.
142
+
143
+ ## Limitations
144
+
145
+ - **Small test set.** 173 commits. Seed 2024 is the default because it scored best of the three
146
+ on them, so its own scores are optimistic; quote the mean.
147
+ - **Clean training data.** Levin & Yehudai dropped commits they could not classify confidently,
148
+ so the model has never seen an ambiguous commit and will still name a class for one. The
149
+ `unclassified` threshold helps, but it was set on that clean data and is probably too
150
+ permissive elsewhere.
151
+ - **Java only.** All training commits come from 11 Java projects.
152
+ - **Weak outside check.** Applied to 34,609 commits from five other Java projects, its predictions
153
+ agreed with the class implied by maintainers' issue labels 58.3% of the time, rising from 37%
154
+ for the least confident commits to 78% for the most. Those labels are themselves noisy.
155
+
156
+ ## Intended use
157
+
158
+ Research on software maintenance: labelling commits at scale by the kind of work they do, as a
159
+ feature or a stratifier. It is not meant for judging individual developers' work.
160
+
161
+ ## Citation
162
+
163
+ If you use this model, please cite the repository (its `CITATION.cff`) and the work it builds on:
164
+
165
+ ```bibtex
166
+ @inproceedings{levin2017boosting,
167
+ author = {Levin, Stanislav and Yehudai, Amiram},
168
+ title = {Boosting Automatic Commit Classification Into Maintenance Activities By
169
+ Utilizing Source Code Changes},
170
+ booktitle = {Proceedings of the 13th International Conference on Predictive Models and
171
+ Data Analytics in Software Engineering (PROMISE)},
172
+ pages = {97--106},
173
+ year = {2017},
174
+ doi = {10.1145/3127005.3127016}
175
+ }
176
+
177
+ @inproceedings{feng2020codebert,
178
+ author = {Feng, Zhangyin and others},
179
+ title = {{CodeBERT}: A Pre-Trained Model for Programming and Natural Languages},
180
+ booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2020},
181
+ year = {2020}
182
+ }
183
+ ```
commit_intent.py ADDED
@@ -0,0 +1,162 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """
2
+ Commit-intent classifier: is a commit corrective, adaptive or perfective?
3
+
4
+ CodeBERT fine-tuned on Levin & Yehudai's 1,151 labelled commits. It reads the commit message and
5
+ the per-file diff, encodes each piece separately, and attention-pools the pieces into one commit
6
+ vector. This module rebuilds exactly the architecture that was trained, so the published weights
7
+ load into it unchanged.
8
+
9
+ from commit_intent import CommitIntent
10
+ clf = CommitIntent.from_pretrained("Binamra00/codebert-commit-intent")
11
+ clf.predict("Fix NPE when the config file is empty",
12
+ {"src/main/java/Config.java": patch_text})
13
+
14
+ Training code and evaluation: github.com/Binamra00/commit-intent-and-refactoring-levels
15
+ """
16
+
17
+ import json
18
+ import re
19
+ from pathlib import Path
20
+
21
+ import torch
22
+ import torch.nn as nn
23
+ from transformers import AutoConfig, AutoModel, AutoTokenizer
24
+
25
+ DEFAULT_REPO = "Binamra00/codebert-commit-intent"
26
+ HUNK = re.compile(r"^@@ .*? @@")
27
+
28
+
29
+ class CommitClassifier(nn.Module):
30
+ """The trained network. Parameter names match the published state dicts."""
31
+
32
+ def __init__(self, base_model, n_labels=3, dropout=0.2):
33
+ super().__init__()
34
+ # Built from the config alone: the fine-tuned encoder weights come from the state dict.
35
+ self.enc = AutoModel.from_config(AutoConfig.from_pretrained(base_model))
36
+ hid = self.enc.config.hidden_size
37
+ self.score = nn.Linear(hid, 1) # attention score per piece
38
+ self.drop = nn.Dropout(dropout)
39
+ self.out = nn.Linear(hid, n_labels)
40
+
41
+ def forward_pieces(self, window_cls, owner):
42
+ """window_cls: (windows, hid) [CLS] vectors; owner: piece index of each window."""
43
+ n = int(owner.max()) + 1
44
+ pv = torch.zeros(n, window_cls.size(1), device=window_cls.device, dtype=window_cls.dtype)
45
+ pv.index_add_(0, owner, window_cls)
46
+ cnt = torch.bincount(owner, minlength=n).to(window_cls.dtype).unsqueeze(1)
47
+ pv = pv / cnt # windows mean-pooled within a piece
48
+ a = self.score(pv).softmax(0) # pieces attention-pooled into a commit
49
+ return self.out(self.drop((pv * a).sum(0)))
50
+
51
+
52
+ def pieces_from_patches(message, files):
53
+ """
54
+ Render a commit the way the model was trained on it.
55
+
56
+ message : the full commit message, as git records it
57
+ files : {path: unified-diff patch} for each changed file -- GitHub's API `patch` field or one
58
+ file's section of `git diff`. Header and hunk lines are dropped; '+' lines become
59
+ ADDED, '-' lines REMOVED. Context lines are ignored.
60
+
61
+ Returns the message piece followed by one piece per file, largest first.
62
+ """
63
+ pieces = [f"<COMMIT_MESSAGE>{(message or '').strip()}</COMMIT_MESSAGE>"]
64
+ rendered = []
65
+ for path, patch in dict(files or {}).items():
66
+ added, removed = [], []
67
+ for ln in (patch or "").split("\n"):
68
+ if HUNK.match(ln) or ln.startswith(("--- ", "+++ ")):
69
+ continue
70
+ if ln.startswith("+"):
71
+ added.append(ln[1:])
72
+ elif ln.startswith("-"):
73
+ removed.append(ln[1:])
74
+ part = [f"<FILE>{path}</FILE>"]
75
+ if removed:
76
+ part.append("<REMOVED>" + "\n".join(removed) + "</REMOVED>")
77
+ if added:
78
+ part.append("<ADDED>" + "\n".join(added) + "</ADDED>")
79
+ rendered.append("\n".join(part))
80
+ rendered.sort(key=len, reverse=True)
81
+ return pieces + rendered
82
+
83
+
84
+ class CommitIntent:
85
+ """Calibrated predictions from a commit's message and diff."""
86
+
87
+ def __init__(self, model, tokenizer, config, device):
88
+ self.model, self.tok, self.cfg, self.device = model, tokenizer, config, device
89
+ self.labels = config["labels"]
90
+
91
+ @classmethod
92
+ def from_pretrained(cls, repo_or_dir=DEFAULT_REPO, seed=None, device=None):
93
+ """Load from a Hugging Face repo id or a local folder holding config.json and the weights."""
94
+ from safetensors.torch import load_file
95
+ local = Path(repo_or_dir)
96
+ if local.is_dir():
97
+ fetch = lambda name: str(local / name)
98
+ else:
99
+ from huggingface_hub import hf_hub_download
100
+ fetch = lambda name: hf_hub_download(repo_or_dir, name)
101
+ config = json.load(open(fetch("config.json"), encoding="utf-8"))
102
+ seed = str(seed or config["default_seed"])
103
+ device = device or ("cuda" if torch.cuda.is_available() else "cpu")
104
+
105
+ model = CommitClassifier(config["base_model"], len(config["labels"]), config["dropout"])
106
+ state = load_file(fetch(config["weights"][seed]))
107
+ missing, unexpected = model.load_state_dict(state, strict=False)
108
+ # Older transformers versions also save a position_ids buffer; nothing else may differ.
109
+ stray = [k for k in missing + unexpected if not k.endswith("position_ids")]
110
+ if stray:
111
+ raise RuntimeError(f"weights do not match the architecture: {stray[:5]}")
112
+ model.to(device).eval()
113
+ return cls(model, AutoTokenizer.from_pretrained(config["base_model"]), config, device)
114
+
115
+ def _windows(self, text):
116
+ ids = self.tok.encode(text, add_special_tokens=False, truncation=False)
117
+ body = self.cfg["max_length"] - 2
118
+ return [ids[i:i + body] for i in range(0, max(len(ids), 1), body)] or [[]]
119
+
120
+ @torch.no_grad()
121
+ def logits(self, pieces, batch=64):
122
+ """Raw logits for one commit given its rendered pieces."""
123
+ pieces = pieces[: self.cfg["max_pieces"]]
124
+ wins, owner = [], []
125
+ for pi, p in enumerate(pieces):
126
+ for w in self._windows(p):
127
+ wins.append(w)
128
+ owner.append(pi)
129
+ cls_vecs = []
130
+ for s in range(0, len(wins), batch):
131
+ chunk = wins[s:s + batch]
132
+ L = max(len(w) for w in chunk) + 2
133
+ ids = torch.full((len(chunk), L), self.tok.pad_token_id, dtype=torch.long)
134
+ att = torch.zeros((len(chunk), L), dtype=torch.long)
135
+ for j, w in enumerate(chunk):
136
+ seq = [self.tok.cls_token_id] + w + [self.tok.sep_token_id]
137
+ ids[j, :len(seq)] = torch.tensor(seq)
138
+ att[j, :len(seq)] = 1
139
+ h = self.model.enc(input_ids=ids.to(self.device),
140
+ attention_mask=att.to(self.device)).last_hidden_state[:, 0]
141
+ cls_vecs.append(h)
142
+ return self.model.forward_pieces(torch.cat(cls_vecs),
143
+ torch.tensor(owner, device=self.device)).cpu()
144
+
145
+ def predict(self, message, files, threshold=None):
146
+ """
147
+ Predicted class with calibrated probabilities.
148
+
149
+ Probabilities use the temperature fitted on held-out commits. `margin` is the top
150
+ probability minus the runner-up; below `threshold` (default 0.47) the commit is reported
151
+ as 'unclassified' rather than forced into a class.
152
+ """
153
+ z = self.logits(pieces_from_patches(message, files)) / self.cfg["temperature"]
154
+ p = z.softmax(-1)
155
+ top2 = p.topk(2).values
156
+ margin = float(top2[0] - top2[1])
157
+ threshold = self.cfg["margin_threshold"] if threshold is None else threshold
158
+ label = self.labels[int(p.argmax())]
159
+ return {"label": label if margin >= threshold else "unclassified",
160
+ "predicted": label,
161
+ "probabilities": {l: round(float(v), 4) for l, v in zip(self.labels, p)},
162
+ "margin": round(margin, 4)}
config.json ADDED
@@ -0,0 +1,21 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architecture": "CommitClassifier",
3
+ "base_model": "microsoft/codebert-base",
4
+ "labels": ["adaptive", "corrective", "perfective"],
5
+ "pooling": "attention",
6
+ "max_length": 512,
7
+ "window_overlap": 0,
8
+ "max_pieces": 256,
9
+ "dropout": 0.2,
10
+ "temperature": 2.86,
11
+ "margin_threshold": 0.47,
12
+ "sensitivity_margin_threshold": 0.34,
13
+ "default_seed": 2024,
14
+ "weights": {
15
+ "42": "model_attention_42.safetensors",
16
+ "1337": "model_attention_1337.safetensors",
17
+ "2024": "model_attention_2024.safetensors"
18
+ },
19
+ "training_code": "https://github.com/Binamra00/commit-intent-and-refactoring-levels",
20
+ "data": "https://doi.org/10.5281/zenodo.24295072"
21
+ }
model_attention_1337.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:84a7e71878176c1e4c0d7f6d5875ea6c37e81849be4584bce3a87ebd90dac84f
3
+ size 498618432
model_attention_2024.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0996bcdf063d9fa71702eeeabb461f40d0550a77c0d8eaa5afe0e0ac7c975f73
3
+ size 498618432
model_attention_42.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ebc1f5f522aea6bc393d5c29650979de807b96b7b3a8782d07fdbe411b26d758
3
+ size 498618424