| --- |
| tags: |
| - text-classification |
| - github-issues |
| library_name: transformers |
| metrics: |
| - f1 |
| - accuracy |
| base_model: distilbert-base-uncased |
| datasets: |
| - pngwn/github-issues-4class |
| license: mit |
| --- |
| |
| # DistilBERT GitHub Issue Classifier (bug / feature / question / support) |
|
|
| Full fine-tune of `distilbert-base-uncased` (66M params): lr 2e-5, batch 16, 4 epochs, weight decay 0.01, 10% warmup, title+body concatenated and truncated to 256 tokens (the truncation used by the NLBSE'24 competition winners). |
|
|
| **Training data:** [pngwn/github-issues-4class](https://huggingface.co/datasets/pngwn/github-issues-4class) — 1,997 balanced issues (bug/feature/question from NLBSE'24 + support class sourced from maintainer-assigned GitHub labels). |
|
|
| ## Test-set results (1,997 balanced examples) |
|
|
| | metric | value | |
| |---|---| |
| | macro-F1 (4-class) | 0.794 | |
| | accuracy | 0.794 | |
| | macro-F1 on bug/feature/question subset | 0.779 | |
| | F1 bug | 0.787 | |
| | F1 feature | 0.788 | |
| | F1 question | 0.710 | |
| | F1 support | 0.891 | |
|
|
| Trails the smaller SetFit MiniLM model ([pngwn/github-issue-classifier-setfit-minilm](https://huggingface.co/pngwn/github-issue-classifier-setfit-minilm), 22M params, 0.807 macro-F1) on the same data — consistent with the NLBSE literature that contrastive SetFit few-shot recipes outperform straight fine-tunes at this scale. |
|
|
| ## Usage |
|
|
| ```python |
| from transformers import AutoModelForSequenceClassification, AutoTokenizer |
| |
| tok = AutoTokenizer.from_pretrained("pngwn/github-issue-classifier-distilbert") |
| model = AutoModelForSequenceClassification.from_pretrained("pngwn/github-issue-classifier-distilbert") |
| inputs = tok("App crashes when opening settings", return_tensors="pt") |
| preds = model(**inputs).logits.argmax(dim=-1) |
| ``` |
|
|
| Labels (id → name): 0 bug, 1 feature, 2 question, 3 support. |
