--- library_name: transformers tags: - vulnerability-reporting - usenix-security-2025 - github-issues - security-issue-classification license: cc language: - en base_model: - FacebookAI/roberta-base pipeline_tag: text-classification --- # USENIX25 — RoBERTa Security Issue Classifier This repository hosts a Hugging Face copy of the fine-tuned **RoBERTa security-issue classifier** associated with the USENIX Security 2025 paper **“I wasn't sure if this is indeed a security risk”: Data-driven Understanding of Security Issue Reporting in GitHub Repositories of Open Source npm Packages** by **Rajdeep Ghosh, Shiladitya De, and Mainack Mondal**. **Paper:** https://arxiv.org/abs/2506.07728 **Original research artifact:** https://doi.org/10.5281/zenodo.15614029 **Hosted checkpoint:** https://huggingface.co/sifat6412/USENIX25-Security-Issue-Classifier > **Attribution:** The model and research are the work of the original paper authors. This repository is a hosted copy, not a claim of original model authorship or independent reproduction of the paper's results. Verify permission to redistribute the weights and the applicable license before making this repository public. A generic `license: cc` value is not sufficient to establish the checkpoint's license. ## Model Details ### Model Description The classifier predicts whether a GitHub issue report is **security-related** based on its title and body. It is a binary sequence-classification model, not a vulnerability verifier. - **Developed by:** Rajdeep Ghosh, Shiladitya De, and Mainack Mondal (original research) - **Shared by:** [sifat6412](https://huggingface.co/sifat6412) (Hugging Face hosting) - **Model type:** Fine-tuned RoBERTa-base sequence classifier, two classes - **Language:** English issue reports (study context) - **License:** Not verified for redistribution; consult original artifact terms - **Fine-tuned from:** [FacebookAI/roberta-base](https://huggingface.co/FacebookAI/roberta-base) ### Model Sources - **Paper:** https://arxiv.org/abs/2506.07728 - **Publication:** 34th USENIX Security Symposium (2025) - **Research artifact:** https://doi.org/10.5281/zenodo.15614029 - **Model repository:** https://huggingface.co/sifat6412/USENIX25-Security-Issue-Classifier ## Uses ### Direct Use Screen GitHub issue titles and descriptions to prioritize issues potentially related to software security. The intended prediction is *security-related issue* versus *non-security-related issue*. ### Downstream Use Research on untagged security reports, issue triage, open-source security reporting, and comparative evaluations on additional repositories or ecosystems. ## Bias, Risks, and Limitations - The original work studied GitHub issue reports from repositories of **npm packages**; transfer to other ecosystems has not been established by the reported evaluation. - Issue reports may be incomplete, ambiguous, or contain external links instead of details. - Security issues without characteristic vocabulary may be missed; unrelated issues with security-related terms may be flagged. - A 512-token inference cutoff can omit information from long issues. - Scores are model probabilities, not independently calibrated estimates of real-world vulnerability prevalence. ### Recommendations Use predictions for triage and empirical analysis, check the label mapping in `config.json`, evaluate on the target repository and class distribution, and manually review consequential cases. ## How to Get Started with the Model Install dependencies: ```bash pip install torch transformers ``` ```python import torch from transformers import AutoTokenizer, AutoModelForSequenceClassification model_id = "sifat6412/USENIX25-Security-Issue-Classifier" tokenizer = AutoTokenizer.from_pretrained(model_id) model = AutoModelForSequenceClassification.from_pretrained(model_id) model.eval() title = "Authentication bypass in API route" body = "An unauthenticated request can access a restricted endpoint." text = f"Issue title: {title} Issue details: {body}" inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=512) with torch.no_grad(): probabilities = torch.softmax(model(**inputs).logits, dim=-1)[0] # Original artifact's inference code maps index 1 to Security Issue. # Confirm this against config.json for the uploaded checkpoint. print("Security-related probability:", probabilities[1].item()) print("Predicted security-related:", probabilities[1].item() >= 0.5) ``` ## Training Details ### Training Data The authors manually labeled **2,100 GitHub issues**, including **1,042 security-related** and **1,058 non-security-related** reports. The paper's larger collection covered **10,907,467 issues** across GitHub repositories associated with **45,466 npm packages**; this full collection is **not** the labeled training set. ### Training Procedure The paper formulated security-issue identification as supervised binary text classification using the issue **title and description**. It compared multiple architectures and selected fine-tuned RoBERTa for subsequent analysis. The authors report systematic hyperparameter optimization using **Optuna** and an **80%/20% training/validation split**. #### Preprocessing At inference, the accompanying artifact uses a single text string containing the issue title and details. The example above preserves this format, with `max_length=512` as used by the supplied evaluation script. This card does not claim the paper specified every tokenizer or preprocessing option. #### Training Hyperparameters The exact learning rate, batch size, epochs, numerical precision, and optimizer settings for this hosted checkpoint are **not independently verified here**. Consult the original code and configuration in the research artifact rather than assuming defaults. #### Speeds, Sizes, Times Not independently benchmarked for this hosted checkpoint; inference times depend on device, batch size, and input length. ## Evaluation ### Testing Data, Factors & Metrics #### Testing Data The paper reports results on its held-out split of manually annotated issue reports, as well as a **separate manual review of 100 previously unseen issues**. #### Factors The reported binary evaluation compares security-related and non-security-related issue classes. #### Metrics Accuracy, macro F1, ROC-AUC, class-wise precision and recall, and an additional manual-check confusion matrix. ### Results **Reported by the original authors (not rerun for this Hugging Face upload):** | Metric | Result | | --- | ---: | | Accuracy | 0.94 | | Macro F1 | 0.94 | | ROC-AUC | 0.94 | | Security-related precision | 0.95 | | Security-related recall | 0.93 | | Non-security precision | 0.93 | | Non-security recall | 0.95 | The independently manually checked 100-issue sample had **96 correct classifications** (6 true security positives, 90 true non-security negatives, 3 missed security issues, and 1 false positive). #### Summary The authors applied their classifier to untagged npm-repository issues and identified **1,617,738 potentially security-related untagged issues**. ## Model Examination The paper reports examining influential tokens using final-layer RoBERTa attention scores. See the paper's appendix for the authors' analysis; attention scores alone should not be treated as causal explanations of predictions. ## Environmental Impact Training hardware, cloud region, duration, and carbon emissions for the uploaded checkpoint have not been independently verified. No numerical carbon estimate is claimed. ## Technical Specifications ### Model Architecture and Objective RoBERTa-base with a two-class sequence-classification head. Input consists of issue title and description, and output is a security-relevance class score. Check the checkpoint's `config.json` for the exact `id2label` mapping. ### Compute Infrastructure #### Hardware Not confirmed from the hosted checkpoint. #### Software Compatible with Hugging Face `transformers` and PyTorch. Original experiments used Optuna for model-selection hyperparameter tuning; precise package versions should be checked in the original artifact. ## Citation **Paper:** Ghosh, Rajdeep, Shiladitya De, and Mainack Mondal. "" I wasn't sure if this is indeed a security risk": Data-driven Understanding of Security Issue Reporting in {GitHub} Repositories of Open Source npm Packages." 34th USENIX Security Symposium (USENIX Security 25). 2025. **BibTeX:** ```bibtex @inproceedings{ghosh2025wasn, title={" I wasn't sure if this is indeed a security risk": Data-driven Understanding of Security Issue Reporting in $\{$GitHub$\}$ Repositories of Open Source npm Packages}, author={Ghosh, Rajdeep and De, Shiladitya and Mondal, Mainack}, booktitle={34th USENIX Security Symposium (USENIX Security 25)}, pages={2145--2164}, year={2025} } ``` ## Glossary - **Security-related issue:** A GitHub issue report classified as concerning a security problem; this does not imply a verified vulnerability. - **False negative:** A genuinely security-related issue incorrectly predicted to be non-security-related. ## More Information See the [original paper](https://arxiv.org/abs/2506.07728), [research artifact](https://doi.org/10.5281/zenodo.15614029), and [hosted checkpoint](https://huggingface.co/sifat6412/USENIX25-Security-Issue-Classifier). ## Model Card Authors Prepared for the `sifat6412/USENIX25-Security-Issue-Classifier` hosting repository, based on the original authors' published research; not an official model card from the paper authors. ## Model Card Contact For original model/research questions, consult the publication and associated artifact. For hosting issues, contact the maintainer through the Hugging Face repository.