JerroldH's picture
Rename Package research checkpoint to Package-SBR
da9821b verified
|
Raw History Blame Contribute Delete
5.09 kB
---
library_name: transformers
pipeline_tag: text-classification
license: apache-2.0
base_model: AlexWortega/openjev
datasets:
- ProtectSkills/MaliciousSkillBench
tags:
- skillscreen
- agent-security
- nli
- source-balanced-random
- train-only
---
# SkillScreen-2B-Package-SBR
2B decision model for `SKILL.md` **and related file contents**, trained on the
recovered SRC010 packages in the original MaliciousSkillBench **SBR training split**,
with a 32K input budget and seed 42.
Formerly `SkillScreen-2B-Package-Experimental`. The repository name changed; weights
and existing `v1` / `v1.1` tags are unchanged. For the latest all-data deployment
release, use [SkillScreen-2B-Package](https://huggingface.co/SkillScreen/SkillScreen-2B-Package).
Based on [OpenJev 2B](https://huggingface.co/AlexWortega/openjev). Full weights are at the root.
The original LoRA adapter and classification head are in `adapter/`.
## Source-checkpoint evaluation
Reference results for source LoRA checkpoint `package-sbr-s42-r001`, seed 42, update 180. These use its original held-out predictions. Validation checks for the merged full-weight export are in [merge_verification.json](merge_verification.json).
- **SRC010 recovered-package test**: 811 skills, 510 malicious and 301 benign, from the original MaliciousSkillBench SBR test partition. Input is SKILL.md plus eligible companion files, with a 32,768-token budget.
- **SkillsBench-1650 external**: 1,650 skills, 150 malicious and 1,500 benign, from `zenith6888/SkillsBench-1650` at revision `68c0d4bf857f48e5fa4f3baa4613f0c0980a7fb2`. Input is the original Markdown plus the provided `script_content` as a companion file, with the same 32K budget.
| Test set | Threshold rule | Native score threshold | Recall (%) | FPR (%) |
| --- | --- | ---: | ---: | ---: |
| SRC010 recovered-package test | Default | 0.5 | 99.80 | 0.66 |
| SRC010 recovered-package test | Target 99% validation recall | 0.99748853624125544 | 99.02 | 0.33 |
| SkillsBench-1650 external | Default | 0.5 | 66.67 | 0.40 |
| SkillsBench-1650 external | Target 99% validation recall | 0.99748853624125544 | 32.67 | 0.13 |
The native score is the entailment softmax; flag scores greater than or equal to the threshold. Both cutoffs were fixed before test and external scoring. The 99% validation target describes threshold selection; the table reports achieved recall on each test set.
## Use
Requires CUDA with BF16 support.
```bash
pip install huggingface_hub
hf download SkillScreen/SkillScreen-2B-Package-SBR --revision v1.1 --local-dir skillscreen
pip install -r skillscreen/requirements.txt
python skillscreen/score.py path/to/package.json
```
Native maliciousness score **≥ 0.5** is flagged as suspicious.
`package.json` contains Markdown and companion file text.
```json
{"skill_text": "Contents of SKILL.md", "files": [{"path": "scripts/helper.py", "text": "File contents"}]}
```
[Checkpoint details](provenance.json) · [Export verification](merge_verification.json) · [Project page](https://jerroldh.github.io/SkillScreen-webpage/) · [Code](https://github.com/JerroldH/SkillScreen)
Jerrold Huang, Junwei Quan, Saiyue Lyu, Zihan Guo, Yingqiang Ge.
## Citation
If you use SkillScreen's code, models, or predictions, or build models based on SkillScreen in your research, please cite the SkillScreen paper for the underlying method. Package-SBR is a separate extension; its package-context results are not reported in that paper. If you release a derived model, please identify SkillScreen as its source and link to the original checkpoint.
```bibtex
@article{huang2026skillscreen,
title = {SkillScreen: High-Recall Screening of Malicious Agent Skills with Decision Models},
author = {Huang, Jerrold and Quan, Junwei and Lyu, Saiyue and Guo, Zihan and Ge, Yingqiang},
journal = {arXiv preprint arXiv:2610.17727},
year = {2026},
doi = {10.48550/arXiv.2610.17727},
url = {https://arxiv.org/abs/2610.17727}
}
```
Machine-readable citation metadata is provided in [CITATION.cff](CITATION.cff). These citation requests do not add conditions to the licenses below.
## License
Model weights (including the LoRA adapter and classification head), project-authored code, configuration, and code examples are licensed under [Apache-2.0](LICENSE), for rights held by the SkillScreen contributors. The model card, documentation, citation metadata, and released provenance/evaluation reports are licensed under [CC BY 4.0](LICENSE-CC-BY-4.0).
Research and commercial use, fine-tuning, and redistribution are permitted under these licenses without individual permission requests. Preserve the applicable attribution and license notices, and identify modifications as required by each license.
Upstream OpenJev and Qwen rights and notices remain applicable. See [THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md), [LICENSE-OPENJEV](LICENSE-OPENJEV), and [LICENSE-QWEN](LICENSE-QWEN).
The `v1.1` revision updates licensing and citation information. Its model weights, adapters, tokenizer, scoring code, and configurations are unchanged from `v1`.