| --- |
| library_name: transformers |
| tags: |
| - vulnerability-reporting |
| - usenix-security-2025 |
| - github-issues |
| - security-issue-classification |
| license: cc |
| language: |
| - en |
| base_model: |
| - FacebookAI/roberta-base |
| pipeline_tag: text-classification |
| --- |
| |
| # USENIX25 — RoBERTa Security Issue Classifier |
|
|
| This repository hosts a Hugging Face copy of the fine-tuned **RoBERTa security-issue classifier** associated with the USENIX Security 2025 paper **“I wasn't sure if this is indeed a security risk”: Data-driven Understanding of Security Issue Reporting in GitHub Repositories of Open Source npm Packages** by **Rajdeep Ghosh, Shiladitya De, and Mainack Mondal**. |
|
|
| **Paper:** https://arxiv.org/abs/2506.07728 |
| **Original research artifact:** https://doi.org/10.5281/zenodo.15614029 |
| **Hosted checkpoint:** https://huggingface.co/sifat6412/USENIX25-Security-Issue-Classifier |
|
|
| > **Attribution:** The model and research are the work of the original paper authors. This repository is a hosted copy, not a claim of original model authorship or independent reproduction of the paper's results. Verify permission to redistribute the weights and the applicable license before making this repository public. A generic `license: cc` value is not sufficient to establish the checkpoint's license. |
|
|
| ## Model Details |
|
|
| ### Model Description |
|
|
| The classifier predicts whether a GitHub issue report is **security-related** based on its title and body. It is a binary sequence-classification model, not a vulnerability verifier. |
|
|
| - **Developed by:** Rajdeep Ghosh, Shiladitya De, and Mainack Mondal (original research) |
| - **Shared by:** [sifat6412](https://huggingface.co/sifat6412) (Hugging Face hosting) |
| - **Model type:** Fine-tuned RoBERTa-base sequence classifier, two classes |
| - **Language:** English issue reports (study context) |
| - **License:** Not verified for redistribution; consult original artifact terms |
| - **Fine-tuned from:** [FacebookAI/roberta-base](https://huggingface.co/FacebookAI/roberta-base) |
|
|
| ### Model Sources |
|
|
| - **Paper:** https://arxiv.org/abs/2506.07728 |
| - **Publication:** 34th USENIX Security Symposium (2025) |
| - **Research artifact:** https://doi.org/10.5281/zenodo.15614029 |
| - **Model repository:** https://huggingface.co/sifat6412/USENIX25-Security-Issue-Classifier |
|
|
| ## Uses |
|
|
| ### Direct Use |
|
|
| Screen GitHub issue titles and descriptions to prioritize issues potentially related to software security. The intended prediction is *security-related issue* versus *non-security-related issue*. |
|
|
| ### Downstream Use |
|
|
| Research on untagged security reports, issue triage, open-source security reporting, and comparative evaluations on additional repositories or ecosystems. |
|
|
| ## Bias, Risks, and Limitations |
|
|
| - The original work studied GitHub issue reports from repositories of **npm packages**; transfer to other ecosystems has not been established by the reported evaluation. |
| - Issue reports may be incomplete, ambiguous, or contain external links instead of details. |
| - Security issues without characteristic vocabulary may be missed; unrelated issues with security-related terms may be flagged. |
| - A 512-token inference cutoff can omit information from long issues. |
| - Scores are model probabilities, not independently calibrated estimates of real-world vulnerability prevalence. |
|
|
| ### Recommendations |
|
|
| Use predictions for triage and empirical analysis, check the label mapping in `config.json`, evaluate on the target repository and class distribution, and manually review consequential cases. |
|
|
| ## How to Get Started with the Model |
|
|
| Install dependencies: |
|
|
| ```bash |
| pip install torch transformers |
| ``` |
|
|
| ```python |
| import torch |
| from transformers import AutoTokenizer, AutoModelForSequenceClassification |
| |
| model_id = "sifat6412/USENIX25-Security-Issue-Classifier" |
| tokenizer = AutoTokenizer.from_pretrained(model_id) |
| model = AutoModelForSequenceClassification.from_pretrained(model_id) |
| model.eval() |
| |
| title = "Authentication bypass in API route" |
| body = "An unauthenticated request can access a restricted endpoint." |
| text = f"Issue title: {title} Issue details: {body}" |
| |
| inputs = tokenizer(text, return_tensors="pt", truncation=True, |
| max_length=512) |
| with torch.no_grad(): |
| probabilities = torch.softmax(model(**inputs).logits, dim=-1)[0] |
| |
| # Original artifact's inference code maps index 1 to Security Issue. |
| # Confirm this against config.json for the uploaded checkpoint. |
| print("Security-related probability:", probabilities[1].item()) |
| print("Predicted security-related:", probabilities[1].item() >= 0.5) |
| ``` |
|
|
| ## Training Details |
|
|
| ### Training Data |
|
|
| The authors manually labeled **2,100 GitHub issues**, including **1,042 security-related** and **1,058 non-security-related** reports. The paper's larger collection covered **10,907,467 issues** across GitHub repositories associated with **45,466 npm packages**; this full collection is **not** the labeled training set. |
|
|
| ### Training Procedure |
|
|
| The paper formulated security-issue identification as supervised binary text classification using the issue **title and description**. It compared multiple architectures and selected fine-tuned RoBERTa for subsequent analysis. The authors report systematic hyperparameter optimization using **Optuna** and an **80%/20% training/validation split**. |
|
|
| #### Preprocessing |
|
|
| At inference, the accompanying artifact uses a single text string containing the issue title and details. The example above preserves this format, with `max_length=512` as used by the supplied evaluation script. This card does not claim the paper specified every tokenizer or preprocessing option. |
|
|
| #### Training Hyperparameters |
|
|
| The exact learning rate, batch size, epochs, numerical precision, and optimizer settings for this hosted checkpoint are **not independently verified here**. Consult the original code and configuration in the research artifact rather than assuming defaults. |
|
|
| #### Speeds, Sizes, Times |
|
|
| Not independently benchmarked for this hosted checkpoint; inference times depend on device, batch size, and input length. |
|
|
| ## Evaluation |
|
|
| ### Testing Data, Factors & Metrics |
|
|
| #### Testing Data |
|
|
| The paper reports results on its held-out split of manually annotated issue reports, as well as a **separate manual review of 100 previously unseen issues**. |
|
|
| #### Factors |
|
|
| The reported binary evaluation compares security-related and non-security-related issue classes. |
|
|
| #### Metrics |
|
|
| Accuracy, macro F1, ROC-AUC, class-wise precision and recall, and an additional manual-check confusion matrix. |
|
|
| ### Results |
|
|
| **Reported by the original authors (not rerun for this Hugging Face upload):** |
|
|
| | Metric | Result | |
| | --- | ---: | |
| | Accuracy | 0.94 | |
| | Macro F1 | 0.94 | |
| | ROC-AUC | 0.94 | |
| | Security-related precision | 0.95 | |
| | Security-related recall | 0.93 | |
| | Non-security precision | 0.93 | |
| | Non-security recall | 0.95 | |
|
|
| The independently manually checked 100-issue sample had **96 correct classifications** (6 true security positives, 90 true non-security negatives, 3 missed security issues, and 1 false positive). |
|
|
| #### Summary |
|
|
| The authors applied their classifier to untagged npm-repository issues and identified **1,617,738 potentially security-related untagged issues**. |
|
|
| ## Model Examination |
|
|
| The paper reports examining influential tokens using final-layer RoBERTa attention scores. See the paper's appendix for the authors' analysis; attention scores alone should not be treated as causal explanations of predictions. |
|
|
| ## Environmental Impact |
|
|
| Training hardware, cloud region, duration, and carbon emissions for the uploaded checkpoint have not been independently verified. No numerical carbon estimate is claimed. |
|
|
| ## Technical Specifications |
|
|
| ### Model Architecture and Objective |
|
|
| RoBERTa-base with a two-class sequence-classification head. Input consists of issue title and description, and output is a security-relevance class score. Check the checkpoint's `config.json` for the exact `id2label` mapping. |
|
|
| ### Compute Infrastructure |
|
|
| #### Hardware |
|
|
| Not confirmed from the hosted checkpoint. |
|
|
| #### Software |
|
|
| Compatible with Hugging Face `transformers` and PyTorch. Original experiments used Optuna for model-selection hyperparameter tuning; precise package versions should be checked in the original artifact. |
|
|
| ## Citation |
|
|
| **Paper:** |
|
|
| Ghosh, Rajdeep, Shiladitya De, and Mainack Mondal. "" I wasn't sure if this is indeed a security risk": Data-driven Understanding of Security Issue Reporting in {GitHub} Repositories of Open Source npm Packages." 34th USENIX Security Symposium (USENIX Security 25). 2025. |
|
|
| **BibTeX:** |
|
|
| ```bibtex |
| @inproceedings{ghosh2025wasn, |
| title={" I wasn't sure if this is indeed a security risk": Data-driven Understanding of Security Issue Reporting in $\{$GitHub$\}$ Repositories of Open Source npm Packages}, |
| author={Ghosh, Rajdeep and De, Shiladitya and Mondal, Mainack}, |
| booktitle={34th USENIX Security Symposium (USENIX Security 25)}, |
| pages={2145--2164}, |
| year={2025} |
| } |
| ``` |
|
|
| ## Glossary |
|
|
| - **Security-related issue:** A GitHub issue report classified as concerning a security problem; this does not imply a verified vulnerability. |
| - **False negative:** A genuinely security-related issue incorrectly predicted to be non-security-related. |
|
|
| ## More Information |
|
|
| See the [original paper](https://arxiv.org/abs/2506.07728), [research artifact](https://doi.org/10.5281/zenodo.15614029), and [hosted checkpoint](https://huggingface.co/sifat6412/USENIX25-Security-Issue-Classifier). |
|
|
| ## Model Card Authors |
|
|
| Prepared for the `sifat6412/USENIX25-Security-Issue-Classifier` hosting repository, based on the original authors' published research; not an official model card from the paper authors. |
|
|
| ## Model Card Contact |
|
|
| For original model/research questions, consult the publication and associated artifact. For hosting issues, contact the maintainer through the Hugging Face repository. |
|
|
