README.md

---
license: apache-2.0
base_model: google-t5/t5-small
tags:
- commit-message-generation
- text2text-generation
- summarization
- code
datasets:
- Maxscha/commitbench
language:
- en
library_name: transformers
pipeline_tag: text-generation
metrics:
- rouge
- bleu
---

# thealper2/t5-small-commitbench

`google-t5/t5-small` fine-tuned on [Maxscha/commitbench](https://huggingface.co/datasets/Maxscha/commitbench) for
commit message generation: given a git diff, generate the commit message describing it.

## Task format

Text-to-text. The input is a task prefix followed by the raw git diff, the target is the
commit message.

```
generate commit message: <git diff>
```

## Training data

[Maxscha/commitbench](https://huggingface.co/datasets/Maxscha/commitbench) official splits, used unchanged:

| Split | Examples in split | Examples used |
|---|---|---|
| train | 1,165,213 | 500,000 |
| validation | 249,689 | 2,000 |
| test | 249,688 | not used for training |

Languages covered by the dataset: Python, JavaScript, PHP, Ruby, Java, Go.

## Training configuration

| Setting | Value |
|---|---|
| Base model | `google-t5/t5-small` |
| Parameters | 60.5M |
| Max source length | 512 tokens |
| Max target length | 64 tokens |
| Per-device batch size | 32 |
| Gradient accumulation | 1 |
| Effective batch size | 32 |
| Learning rate | 3e-05 |
| LR schedule | linear |
| Warmup ratio | 0.05 |
| Weight decay | 0.01 |
| Epochs | 2.0 |
| Label smoothing | 0.0 |
| Gradient clipping | 1.0 |
| Mixed precision | bf16 |
| Seed | 42 |
| Optimizer | AdamW |
| Training time | 1.219 h |
| Hardware | NVIDIA GeForce RTX 5060 Ti (15.9 GB) |

Truncation at these limits (measured on a 50k sample with the T5 tokenizer):
- 0.7% of the diffs exceed 512 source tokens.
- 4.47% of the commit messages exceed 64 target tokens.

## Results

- Final training loss: **3.5762**
- Best validation loss: **3.2414**

Test split (20,000 examples), beam search with `num_beams=4`:

| Metric | Value |
|---|---|
| rouge1 | 19.31 |
| rouge2 | 4.668 |
| rougeL | 17.42 |
| rougeLsum | 17.42 |
| bleu | 2.148 |
| exact_match | 0.04 |
| gen_len_words_mean | 5.005 |
| ref_len_words_mean | 11.27 |

Per programming language:

| Language | n | ROUGE-1 | ROUGE-2 | ROUGE-L | BLEU | Exact match |
|---|---|---|---|---|---|---|
| Python | 5,722 | 21.20 | 6.05 | 19.29 | 2.73 | 0.04 |
| JavaScript | 4,468 | 18.86 | 4.05 | 17.07 | 2.01 | 0.02 |
| PHP | 3,489 | 17.04 | 3.46 | 15.29 | 1.64 | 0.09 |
| Ruby | 2,808 | 22.08 | 5.79 | 19.65 | 2.44 | 0.04 |
| Java | 1,799 | 15.19 | 2.61 | 13.58 | 1.02 | 0.06 |
| Go | 1,714 | 18.65 | 4.45 | 16.75 | 2.11 | 0.00 |

ROUGE and BLEU are lexical-overlap metrics. They do not fully capture whether a commit
message describes a change correctly, and generic messages can score well.

## Usage

```python
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

model_id = "thealper2/t5-small-commitbench"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSeq2SeqLM.from_pretrained(model_id)

diff = open("change.patch").read()
inputs = tokenizer(
    "generate commit message: " + diff,
    max_length=512,
    truncation=True,
    return_tensors="pt",
)
output = model.generate(
    **inputs,
    num_beams=4,
    max_new_tokens=64,
    length_penalty=1.0,
    no_repeat_ngram_size=3,
)
print(tokenizer.decode(output[0], skip_special_tokens=True))
```

Default generation settings: `num_beams=4`, `max_new_tokens=64`,
`min_new_tokens=0`, `length_penalty=1.0`,
`no_repeat_ngram_size=3`, `do_sample=False` (deterministic).

## Limitations

- CommitBench replaces identifying literals with placeholder tokens: every diff contains
  `<HASH>` instead of commit hashes, and 26.5% of the reference messages contain `<I>` (numbers),
  `<URL>` or `<EMAIL>`. The model therefore also generates these tokens, e.g. `Bumped version to <I>`.
- The T5 sentencepiece vocabulary does not cover every character used in source code (curly braces, backslashes, angle brackets), so about 2.35% of the input tokens become `<unk>`. This limits how precisely the model can read a diff.
- Diffs longer than 512 tokens are truncated; the tail of the change is
  not visible to the model.
- CommitBench splits are random over commits, not over repositories: 98.6% of the test examples
  come from repositories that also appear in the training split. No `(diff, message)` pair is
  shared across splits, but the reported scores partly reflect familiarity with a project's
  commit style rather than generalization to unseen code.
- The dataset is English-only and covers six languages; behaviour on other languages or
  on very large multi-file changes is untested.
- CommitBench is released under CC BY-NC 4.0, which restricts commercial use of the data.

## Reproducibility

- python: `3.12.3`
- torch: `2.11.0+cu128`
- transformers: `5.17.0`
- datasets: `4.3.0`
- tokenizers: `0.23.2`
- seed: `42`