|
Download README.md from Binamra00/codebert-commit-intent: direct link, hf CLI and curl.
- Browser
- Download file 7.74 kB
-
https://huggingface.co/Binamra00/codebert-commit-intent/resolve/main/README.md
- Command line
-
hf download hf://Binamra00/codebert-commit-intent/README.md
-
curl -L -o README.md https://huggingface.co/Binamra00/codebert-commit-intent/resolve/main/README.md
7.74 kB
| license: mit | |
| language: en | |
| library_name: pytorch | |
| pipeline_tag: text-classification | |
| base_model: microsoft/codebert-base | |
| tags: | |
| - code | |
| - commit-classification | |
| - maintenance-activities | |
| - software-engineering | |
| - mining-software-repositories | |
| - codebert | |
| - java | |
| metrics: | |
| - accuracy | |
| model-index: | |
| - name: codebert-commit-intent | |
| results: | |
| - task: | |
| type: text-classification | |
| name: Commit maintenance-activity classification | |
| dataset: | |
| type: levin-yehudai-commits | |
| name: Levin & Yehudai labelled commits (173-commit test split) | |
| metrics: | |
| - type: accuracy | |
| value: 86.7 | |
| name: Accuracy, mean of 3 seeds | |
| - type: cohen_kappa | |
| value: 0.791 | |
| name: Cohen's kappa, mean of 3 seeds | |
| # CodeBERT commit-intent classifier | |
| Classifies a commit as **corrective** (fixing a fault), **adaptive** (adding a feature) or | |
| **perfective** (improving the code or its design), Swanson's maintenance categories, from its | |
| message and its diff. | |
| It is [CodeBERT](https://huggingface.co/microsoft/codebert-base) fine-tuned on the 1,151 commits | |
| Levin & Yehudai labelled by hand. Where their method reads 48 counts of AST change types extracted | |
| by ChangeDistiller, this model reads the diff itself, so it runs on modern Java that ChangeDistiller | |
| cannot parse. On the same 173 held-out commits it reaches **86.7% accuracy and Cohen's κ 0.791**, | |
| against 75.7% and 0.621 for a reproduction of their best model. | |
| | | | | |
| |---|---| | |
| | Training code and evaluation | [github.com/Binamra00/commit-intent-and-refactoring-levels](https://github.com/Binamra00/commit-intent-and-refactoring-levels) | | |
| | Data, stage outputs and 34,609 classified commits | [10.5281/zenodo.24295072](https://doi.org/10.5281/zenodo.24295072) | | |
| | Base model | [microsoft/codebert-base](https://huggingface.co/microsoft/codebert-base) | | |
| ## Files | |
| | file | | | |
| |---|---| | |
| | `model_attention_2024.safetensors` | the default model, seed 2024 | | |
| | `model_attention_42.safetensors`, `model_attention_1337.safetensors` | the other two seeds; the reported scores are the mean of all three | | |
| | `config.json` | labels, input limits, calibration temperature and thresholds | | |
| | `commit_intent.py` | the architecture and an inference wrapper; the weights load into it unchanged | | |
| ## Usage | |
| ```bash | |
| pip install torch transformers safetensors huggingface_hub | |
| ``` | |
| ```python | |
| from huggingface_hub import hf_hub_download | |
| import importlib.util | |
| path = hf_hub_download("Binamra00/codebert-commit-intent", "commit_intent.py") | |
| spec = importlib.util.spec_from_file_location("commit_intent", path) | |
| ci = importlib.util.module_from_spec(spec); spec.loader.exec_module(ci) | |
| clf = ci.CommitIntent.from_pretrained("Binamra00/codebert-commit-intent") # seed=42 / 1337 for the others | |
| patch = """@@ -10,7 +10,9 @@ public Config load(Path file) { | |
| - return parse(Files.readString(file)); | |
| + String text = Files.readString(file); | |
| + if (text.isBlank()) return Config.empty(); | |
| + return parse(text); | |
| """ | |
| clf.predict("Fix NPE when the config file is empty", {"src/main/java/Config.java": patch}) | |
| # {'label': ..., 'predicted': ..., 'probabilities': {...}, 'margin': ...} | |
| ``` | |
| **Input.** The full commit message, and for each changed file its unified-diff patch, as GitHub's | |
| API returns it in the `patch` field or as one file's section of `git diff`. Context lines are | |
| ignored. The model was trained on Java projects. | |
| **Output.** | |
| - `probabilities`: calibrated with temperature scaling (T = 2.86). | |
| - `margin`: the top probability minus the runner-up. | |
| - `predicted`: the most likely class. | |
| - `label`: the same class, or `unclassified` when the margin is below 0.47. Pass `threshold=` to | |
| change that cut, or read `predicted` to always get one of the three classes. | |
| ## How it works | |
| Each commit is split into pieces: one for the message, and one per changed file, holding that | |
| file's removed and added lines. Each piece is encoded by CodeBERT, in 512-token windows when it is | |
| longer. Windows are averaged within a piece, and a learned attention score weights the pieces into | |
| one commit vector. A linear layer then predicts the three classes. A commit is never truncated: up | |
| to 256 pieces are kept, and no commit in the training data had more. | |
| ## Training | |
| | | | | |
| |---|---| | |
| | Data | Levin & Yehudai's 1,151 labelled commits from 11 Java projects ([10.5281/zenodo.835534](https://doi.org/10.5281/zenodo.835534)), with diffs fetched from the GitHub API; 1,150 used, one excluded because the API truncated it or it changed no Java file | | |
| | Split | their protocol: 85/15 stratified, 977 training / 173 test commits | | |
| | Model selection | attention vs transformer pooling, by 5-fold cross-validation repeated twice on the training commits | | |
| | Optimiser | AdamW, learning rate 2e-5 for the encoder and 1e-3 for the head, linear warmup over 10% of steps then linear decay, weight decay 0.01, gradient clipping at 1.0 | | |
| | Loss | cross-entropy weighted by inverse class frequency | | |
| | Epochs | up to 10, early stopping after 3 without a better validation κ, on 10% of the training commits | | |
| | Seeds | 42, 1337, 2024 | | |
| | Hardware | one NVIDIA A100 80 GB, bfloat16 autocast | | |
| ## Evaluation | |
| On the 173 test commits, which were opened once, after training: | |
| | seed | accuracy | κ | macro F1 | | |
| |---|---|---|---| | |
| | 42 | 85.5% | 0.771 | 83.4 | | |
| | 1337 | 86.7% | 0.792 | 85.3 | | |
| | 2024 | 87.9% | 0.810 | 86.5 | | |
| | **mean** | **86.7% ± 1.2** | **0.791 ± 0.020** | **85.1** | | |
| For comparison, on the same commits: | |
| | model | accuracy | κ | | |
| |---|---|---| | |
| | Levin & Yehudai, as published | 76.7% | 0.635 | | |
| | Our reproduction of their model | 75.7% | 0.621 | | |
| | Best reproduced feature-based model | 78.0% | 0.654 | | |
| Temperature scaling, fitted on the test commits, took expected calibration error from 0.109 to | |
| 0.025. | |
| ## Limitations | |
| - **Small test set.** 173 commits. Seed 2024 is the default because it scored best of the three | |
| on them, so its own scores are optimistic; quote the mean. | |
| - **Clean training data.** Levin & Yehudai dropped commits they could not classify confidently, | |
| so the model has never seen an ambiguous commit and will still name a class for one. The | |
| `unclassified` threshold helps, but it was set on that clean data and is probably too | |
| permissive elsewhere. | |
| - **Java only.** All training commits come from 11 Java projects. | |
| - **Weak outside check.** Applied to 34,609 commits from five other Java projects, its predictions | |
| agreed with the class implied by maintainers' issue labels 58.3% of the time, rising from 37% | |
| for the least confident commits to 78% for the most. Those labels are themselves noisy. | |
| ## Intended use | |
| Research on software maintenance: labelling commits at scale by the kind of work they do, as a | |
| feature or a stratifier. It is not meant for judging individual developers' work. | |
| ## Citation | |
| If you use this model, please cite the repository (its `CITATION.cff`) and the work it builds on: | |
| ```bibtex | |
| @inproceedings{levin2017boosting, | |
| author = {Levin, Stanislav and Yehudai, Amiram}, | |
| title = {Boosting Automatic Commit Classification Into Maintenance Activities By | |
| Utilizing Source Code Changes}, | |
| booktitle = {Proceedings of the 13th International Conference on Predictive Models and | |
| Data Analytics in Software Engineering (PROMISE)}, | |
| pages = {97--106}, | |
| year = {2017}, | |
| doi = {10.1145/3127005.3127016} | |
| } | |
| @inproceedings{feng2020codebert, | |
| author = {Feng, Zhangyin and others}, | |
| title = {{CodeBERT}: A Pre-Trained Model for Programming and Natural Languages}, | |
| booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2020}, | |
| year = {2020} | |
| } | |
| ``` | |