| # 🧠 Scikit-learn GitHub Issues – Multilabel Classifier |
|
|
| This repository contains a **multilabel text classification model** trained to predict GitHub issue labels for the **scikit-learn** project based on issue text and comments. |
|
|
| The model is suitable for: |
| - automated issue triage |
| - label recommendation |
| - downstream semantic search and filtering pipelines |
|
|
| --- |
|
|
| ## 🔍 Task |
|
|
| **Multilabel Text Classification** |
|
|
| Each GitHub issue can have **multiple labels** (e.g. `Bug`, `Documentation`, `module:linear_model`). |
| The model predicts **all relevant labels** for a given issue text. |
|
|
| --- |
|
|
| ## 📦 Dataset |
|
|
| - **Source**: GitHub Issues from the `scikit-learn/scikit-learn` repository |
| - **Collection method**: Custom GitHub REST API pipeline |
| - **Preprocessing steps**: |
| - Included **open and closed issues** |
| - Excluded **pull requests** |
| - Retrieved **all issue comments** |
| - Exploded comments so each sample contains: |
| - issue title |
| - issue body |
| - comments |
| - Converted labels to **multi-hot vectors** |
|
|
| - **Dataset on Hugging Face**: |
| 👉 https://huggingface.co/datasets/Talip7/scikit-learn-issues-multilabel |
|
|
| **Final dataset size**: ~12,000 samples |
| **Number of unique labels**: ~20+ |
|
|
| --- |
|
|
| ## 🧱 Model |
|
|
| - **Base model**: `distilbert-base-uncased` |
| - **Architecture**: `AutoModelForSequenceClassification` |
| - **Problem type**: `multi_label_classification` |
| - **Loss function**: Binary Cross Entropy with Logits |
| - **Activation**: Sigmoid |
| - **Prediction threshold**: 0.5 |
|
|
| --- |
|
|
| ## 📊 Evaluation Metrics |
|
|
| | Metric | Score | |
| |-----------|-------| |
| | Micro F1 | **0.857** | |
| | Macro F1 | 0.263 | |
| | Epochs | 3 | |
|
|
| **Notes**: |
| - Micro F1 reflects strong overall performance. |
| - Lower Macro F1 is expected due to **severe label imbalance**, common in real-world GitHub issue datasets. |
|
|
| --- |
|
|
| ## 🧪 Training Details |
|
|
| - Optimizer: AdamW |
| - Learning rate: 2e-5 |
| - Batch size: 16 |
| - Max sequence length: 256 |
| - Validation split: 10% |
| - Best model selection: micro-F1 |
| - Trained on GPU |
|
|
| --- |
|
|
| ## 🚀 Inference Example |
|
|
| ```python |
| from transformers import pipeline |
| |
| classifier = pipeline( |
| "text-classification", |
| model="Talip7/scikit-learn-multilabel-classifier", |
| return_all_scores=True |
| ) |
| |
| text = """ |
| Bug occurs in LinearRegression when sample_weight is used. |
| The issue happens after upgrading numpy. |
| """ |
| |
| outputs = classifier(text) |
| |
| labels = [o["label"] for o in outputs[0] if o["score"] > 0.5] |
| print(labels) |
| ``` |
|
|
| --- |
|
|
| ## 🔗 Intended Use |
|
|
| Automated GitHub issue labeling |
|
|
| Developer productivity tools |
|
|
| Search and recommendation systems |
|
|
| Foundation for semantic search + classification pipelines |
|
|
| --- |
|
|
| ## ⚠️ Limitations |
|
|
| Rare labels have limited representation |
|
|
| Threshold-based predictions may require tuning per use case |
|
|
| Model is domain-specific to scikit-learn GitHub issues |
|
|
| --- |
|
|
| ## 🛣️ Future Work |
|
|
| Joint semantic search + multilabel prediction |
|
|
| --- |
|
|
| ## 👤 Author |
|
|
| Talip7 |
