| --- |
| license: apache-2.0 |
| base_model: Qwen/Qwen2.5-Coder-1.5B-Instruct |
| library_name: mlx |
| pipeline_tag: text-generation |
| language: [en] |
| tags: [git, commit-messages, code, lora, mlx, gguf] |
| --- |
| |
| # Commit Suggestor (v2) |
|
|
| Writes the subject line of a git commit from its diff. A LoRA fine-tune of |
| [Qwen2.5-Coder-1.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Coder-1.5B-Instruct), |
| first supervised (SFT) and then preference-tuned (DPO), shipped as MLX 4-bit |
| for Apple Silicon and GGUF Q4_K_M for llama.cpp on any CPU. |
|
|
| Code, data pipeline and every evaluation file: |
| [github.com/nihardamera/Commit-Suggestor](https://github.com/nihardamera/Commit-Suggestor). |
|
|
| ## Use |
|
|
| ```console |
| $ pipx install "commit-suggestor[mlx] @ git+https://github.com/nihardamera/Commit-Suggestor" |
| $ # off a Mac: commit-suggestor[cpu] instead of [mlx] |
| $ commit-suggestor download |
| $ git add -p && commit-suggestor suggest |
| ``` |
|
|
| `commit-suggestor install-hook` pre-fills the message on `git commit`, with a |
| 20-second limit (`COMMIT_SUGGESTOR_TIMEOUT`) and never a download inside a |
| commit. The hook loads the model afresh for each commit, so it waits several |
| seconds on a Mac and more on a CPU. The prompt is |
| `commit_suggestor/prompt.py` (prompt version 2, without repository context); |
| asking the model any other way is testing a prompt it was not trained on. |
| Subjects over 72 characters are shortened at a clause boundary |
| (`prompt.fit_subject`), as in every number below marked "guarded". |
|
|
| ## Results |
|
|
| 450 test commits from three projects never seen in training |
| (django-rest-framework, pydantic, sphinx). The reference is the author's own |
| subject. Means with 95% bootstrap intervals, resampled within each project. |
| Shipped artefacts as users run them: MLX 4-bit, greedy, guarded. |
|
|
| | System | ROUGE-L | Specific % | Generic % | Names not in diff % | Well-formed % | Words | |
| | --- | ---: | ---: | ---: | ---: | ---: | ---: | |
| | Qwen2.5-Coder-1.5B-Instruct, prompt only | 21.09 [19.43, 22.76] | 92.44 [90.0, 94.89] | 0.0 [0.0, 0.0] | 0.44 [0.0, 1.11] | 92.22 [89.56, 94.67] | 7.18 [6.98, 7.38] | |
| | v1: QLoRA in MLX, 2,400 examples | 31.06 [28.89, 33.2] | 80.67 [77.11, 84.22] | 6.22 [4.0, 8.67] | 0.44 [0.0, 1.11] | 98.89 [97.78, 99.78] | 4.71 [4.54, 4.89] | |
| | SFT-v2: LoRA, one epoch of the v2 data | 30.16 [27.97, 32.39] | 79.33 [75.56, 82.89] | 8.44 [6.0, 11.11] | 1.11 [0.22, 2.22] | 100.0 [100.0, 100.0] | 4.42 [4.28, 4.58] | |
| | **this model**: SFT-v2 + DPO (beta 0.3) | 31.13 [29.06, 33.2] | 95.56 [93.56, 97.33] | 0.0 [0.0, 0.0] | 1.56 [0.44, 2.89] | 100.0 [100.0, 100.0] | 6.28 [6.1, 6.46] | |
|
|
| DPO against SFT-v2, paired on the same diffs: specific |
| 16.22 [12.67, 19.78] points, generic -8.44 [-11.11, -6.0], ROUGE-L |
| 0.97 [-0.79, 2.75], names not in the diff 0.44 [-0.67, 1.78], length |
| 1.85 [1.66, 2.04] words. |
|
|
| **LLM judge** (Claude Haiku 5.5, pairwise, both orders; a split verdict is a tie): |
|
|
| | Comparison | Judge | Wins / losses / ties | Win rate, ties excluded | |
| | --- | --- | --- | --- | |
| | this model vs SFT-v2 | rubric v1 (also labelled the DPO pairs) | 268 / 54 / 128 | 83.2% [78.8, 86.9] | |
| | this model vs SFT-v2 | rubric v2 (length-neutral; labelled no pairs) | 291 / 48 / 111 | 85.8% [81.7, 89.2] | |
| | this model vs v1 | rubric v1 | 267 / 74 / 109 | 78.3% [73.6, 82.3] | |
|
|
| The judge agrees with one human rater (the author) only moderately: Cohen's |
| kappa 0.232 on held-out labels, above ROUGE-L's but far from the 0.6 |
| aimed for. Read the win rates as a strong signal from a noisy instrument, and |
| the second rubric as a check that DPO did not just learn the labelling |
| judge's taste. Both judges are the same model, so they share its biases. |
|
|
| **Specific** means the subject names a file or symbol changed in the diff; |
| **generic** means it would fit almost any diff, or was given to three or more |
| diffs. Both are rule-based (`commit_suggestor/metrics.py`), and the specificity |
| rule failed a check against human ratings in v1, so treat it as a proxy. |
|
|
| ## Training |
|
|
| - **Data** ([data card](https://github.com/nihardamera/Commit-Suggestor/blob/main/docs/DATA_CARD.md)): |
| commits mined from 17 other Python projects, filtered (bots, |
| merges, releases, huge diffs, trivial subjects), near-duplicates removed with |
| MinHash (Jaccard >= 0.8), capped at |
| 4,000 per project. Split by project, never by commit. |
| - **SFT-v2**: LoRA r 8, alpha 160, the top 16 of 28 layers, all 7 linear |
| projections (5.28M trainable parameters, 0.34%), learning rate 5e-5, one |
| epoch, loss on the subject only. TRL on a Kaggle T4. More data helped on |
| validation: ROUGE-L 26.8 [24.86, 28.74] at 2,400 |
| examples, 30.1 [28.12, 32.13] at the full epoch. |
| - **DPO**: 4 samples per training diff (T 0.9), ranked by the judge in both |
| orders; 2,198 pairs kept where both orders agreed and the winner neither |
| invented names nor was generic. Chosen subjects are only |
| 0.38 [0.27, 0.5] words longer than rejected ones. One epoch, |
| learning rate 2e-5, sigmoid loss, beta in {0.05, 0.1, 0.3}; |
| 2.1 h on a T4 for all three. The loss was also |
| written by hand (`commit_suggestor/dpo.py`) and matches TRL's on trained |
| models within 0.0010. |
| - **Beta** was chosen on 300 validation diffs by a rule fixed in code before |
| the results were seen: Each beta is judged against beta 0 (SFT-v2, fp16, same job) on the 300 validation diffs. A beta is eligible only if, compared with beta 0, its rate of names not in the diff and its generic rate are each at most 1 point higher, and its mean length is at most 2 words longer, since length is the first thing DPO exploits. The eligible beta with the highest judge win rate (ties excluded) is chosen. It ships instead of SFT-v2 only if the lower end of that win rate's 95% Wilson interval is above 50%. Result: beta 0.3. |
| - **Repository context** (the last five subjects as extra input) was trained in |
| but not shipped: on validation the judge preferred subjects without it |
| (27.3% [21.0, 34.7] win rate for context). |
|
|
| ## Conversion and serving |
|
|
| - MLX 4-bit gives the same subject as fp16 in only |
| 19.56% [16.0, 23.11] of diffs (greedy decoding |
| flips on near-ties); quantisation costs no quality: ROUGE-L against fp16 |
| 1.29 [0.05, 2.55] points. |
| - GGUF Q4_K_M (986 MB, llama.cpp b11701): |
| same subject as fp16 in 25.11% [21.33, 29.11]; |
| ROUGE-L against fp16 2.27 [1.16, 3.41] points. |
| 7.73 s per subject on 4 Kaggle CPU cores, |
| after loading. On a slow CPU the hook can pass its 20-second limit; raise it |
| with `COMMIT_SUGGESTOR_TIMEOUT`. |
| - MLX on an Apple M4, 16 GB: 2.68 s to load, |
| then 0.433 s per subject (p95 0.978 s), |
| 1,446 MB peak memory. |
|
|
| ## Limitations |
|
|
| - Trained and tested on Python projects with English subjects. Other |
| languages and commit conventions are untested. |
| - Subjects only: no message body. |
| - DPO made subjects longer (6.28 words against |
| 4.42), and unguarded it writes past 72 |
| characters in 5.6% of test |
| subjects; the guard cuts those. |
| - It names an identifier that is not in the diff in |
| 1.56% of test subjects |
| (the authors' own subjects: about 5%). Read the suggestion before committing it. |
| - Human preference for DPO over SFT-v2 has not been measured yet. |
|
|
| ## Privacy |
|
|
| The CLI runs locally and sends nothing anywhere. The opt-in feedback log |
| (accepted / edited / replaced) stays on your machine. The demo Space is a static |
| page of stored test-set subjects: it runs no model and takes no input. |
|
|
