File size: 7,742 Bytes
a007f4a | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 | ---
license: mit
language: en
library_name: pytorch
pipeline_tag: text-classification
base_model: microsoft/codebert-base
tags:
- code
- commit-classification
- maintenance-activities
- software-engineering
- mining-software-repositories
- codebert
- java
metrics:
- accuracy
model-index:
- name: codebert-commit-intent
results:
- task:
type: text-classification
name: Commit maintenance-activity classification
dataset:
type: levin-yehudai-commits
name: Levin & Yehudai labelled commits (173-commit test split)
metrics:
- type: accuracy
value: 86.7
name: Accuracy, mean of 3 seeds
- type: cohen_kappa
value: 0.791
name: Cohen's kappa, mean of 3 seeds
---
# CodeBERT commit-intent classifier
Classifies a commit as **corrective** (fixing a fault), **adaptive** (adding a feature) or
**perfective** (improving the code or its design), Swanson's maintenance categories, from its
message and its diff.
It is [CodeBERT](https://huggingface.co/microsoft/codebert-base) fine-tuned on the 1,151 commits
Levin & Yehudai labelled by hand. Where their method reads 48 counts of AST change types extracted
by ChangeDistiller, this model reads the diff itself, so it runs on modern Java that ChangeDistiller
cannot parse. On the same 173 held-out commits it reaches **86.7% accuracy and Cohen's κ 0.791**,
against 75.7% and 0.621 for a reproduction of their best model.
| | |
|---|---|
| Training code and evaluation | [github.com/Binamra00/commit-intent-and-refactoring-levels](https://github.com/Binamra00/commit-intent-and-refactoring-levels) |
| Data, stage outputs and 34,609 classified commits | [10.5281/zenodo.24295072](https://doi.org/10.5281/zenodo.24295072) |
| Base model | [microsoft/codebert-base](https://huggingface.co/microsoft/codebert-base) |
## Files
| file | |
|---|---|
| `model_attention_2024.safetensors` | the default model, seed 2024 |
| `model_attention_42.safetensors`, `model_attention_1337.safetensors` | the other two seeds; the reported scores are the mean of all three |
| `config.json` | labels, input limits, calibration temperature and thresholds |
| `commit_intent.py` | the architecture and an inference wrapper; the weights load into it unchanged |
## Usage
```bash
pip install torch transformers safetensors huggingface_hub
```
```python
from huggingface_hub import hf_hub_download
import importlib.util
path = hf_hub_download("Binamra00/codebert-commit-intent", "commit_intent.py")
spec = importlib.util.spec_from_file_location("commit_intent", path)
ci = importlib.util.module_from_spec(spec); spec.loader.exec_module(ci)
clf = ci.CommitIntent.from_pretrained("Binamra00/codebert-commit-intent") # seed=42 / 1337 for the others
patch = """@@ -10,7 +10,9 @@ public Config load(Path file) {
- return parse(Files.readString(file));
+ String text = Files.readString(file);
+ if (text.isBlank()) return Config.empty();
+ return parse(text);
"""
clf.predict("Fix NPE when the config file is empty", {"src/main/java/Config.java": patch})
# {'label': ..., 'predicted': ..., 'probabilities': {...}, 'margin': ...}
```
**Input.** The full commit message, and for each changed file its unified-diff patch, as GitHub's
API returns it in the `patch` field or as one file's section of `git diff`. Context lines are
ignored. The model was trained on Java projects.
**Output.**
- `probabilities`: calibrated with temperature scaling (T = 2.86).
- `margin`: the top probability minus the runner-up.
- `predicted`: the most likely class.
- `label`: the same class, or `unclassified` when the margin is below 0.47. Pass `threshold=` to
change that cut, or read `predicted` to always get one of the three classes.
## How it works
Each commit is split into pieces: one for the message, and one per changed file, holding that
file's removed and added lines. Each piece is encoded by CodeBERT, in 512-token windows when it is
longer. Windows are averaged within a piece, and a learned attention score weights the pieces into
one commit vector. A linear layer then predicts the three classes. A commit is never truncated: up
to 256 pieces are kept, and no commit in the training data had more.
## Training
| | |
|---|---|
| Data | Levin & Yehudai's 1,151 labelled commits from 11 Java projects ([10.5281/zenodo.835534](https://doi.org/10.5281/zenodo.835534)), with diffs fetched from the GitHub API; 1,150 used, one excluded because the API truncated it or it changed no Java file |
| Split | their protocol: 85/15 stratified, 977 training / 173 test commits |
| Model selection | attention vs transformer pooling, by 5-fold cross-validation repeated twice on the training commits |
| Optimiser | AdamW, learning rate 2e-5 for the encoder and 1e-3 for the head, linear warmup over 10% of steps then linear decay, weight decay 0.01, gradient clipping at 1.0 |
| Loss | cross-entropy weighted by inverse class frequency |
| Epochs | up to 10, early stopping after 3 without a better validation κ, on 10% of the training commits |
| Seeds | 42, 1337, 2024 |
| Hardware | one NVIDIA A100 80 GB, bfloat16 autocast |
## Evaluation
On the 173 test commits, which were opened once, after training:
| seed | accuracy | κ | macro F1 |
|---|---|---|---|
| 42 | 85.5% | 0.771 | 83.4 |
| 1337 | 86.7% | 0.792 | 85.3 |
| 2024 | 87.9% | 0.810 | 86.5 |
| **mean** | **86.7% ± 1.2** | **0.791 ± 0.020** | **85.1** |
For comparison, on the same commits:
| model | accuracy | κ |
|---|---|---|
| Levin & Yehudai, as published | 76.7% | 0.635 |
| Our reproduction of their model | 75.7% | 0.621 |
| Best reproduced feature-based model | 78.0% | 0.654 |
Temperature scaling, fitted on the test commits, took expected calibration error from 0.109 to
0.025.
## Limitations
- **Small test set.** 173 commits. Seed 2024 is the default because it scored best of the three
on them, so its own scores are optimistic; quote the mean.
- **Clean training data.** Levin & Yehudai dropped commits they could not classify confidently,
so the model has never seen an ambiguous commit and will still name a class for one. The
`unclassified` threshold helps, but it was set on that clean data and is probably too
permissive elsewhere.
- **Java only.** All training commits come from 11 Java projects.
- **Weak outside check.** Applied to 34,609 commits from five other Java projects, its predictions
agreed with the class implied by maintainers' issue labels 58.3% of the time, rising from 37%
for the least confident commits to 78% for the most. Those labels are themselves noisy.
## Intended use
Research on software maintenance: labelling commits at scale by the kind of work they do, as a
feature or a stratifier. It is not meant for judging individual developers' work.
## Citation
If you use this model, please cite the repository (its `CITATION.cff`) and the work it builds on:
```bibtex
@inproceedings{levin2017boosting,
author = {Levin, Stanislav and Yehudai, Amiram},
title = {Boosting Automatic Commit Classification Into Maintenance Activities By
Utilizing Source Code Changes},
booktitle = {Proceedings of the 13th International Conference on Predictive Models and
Data Analytics in Software Engineering (PROMISE)},
pages = {97--106},
year = {2017},
doi = {10.1145/3127005.3127016}
}
@inproceedings{feng2020codebert,
author = {Feng, Zhangyin and others},
title = {{CodeBERT}: A Pre-Trained Model for Programming and Natural Languages},
booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2020},
year = {2020}
}
```
|