--- library_name: transformers pipeline_tag: text-classification license: apache-2.0 base_model: AlexWortega/openjev base_model_relation: finetune datasets: - SkillScreen/SkillScreen-Data tags: - skillscreen - agent-security - decision-model - nli - arxiv:2610.17727 --- # SkillScreen-2B-Package **Recommended deployment release v0.1.0 · 32K context** SkillScreen-2B-Package screens AI-agent skills using `SKILL.md` and supporting text files. It returns a maliciousness score and a review decision in one forward pass. Complete merged weights, the original LoRA adapter and frozen operating points are included. [Package model](https://huggingface.co/SkillScreen/SkillScreen-2B-Package) · [Markdown model](https://huggingface.co/SkillScreen/SkillScreen-2B-Markdown) · [Data](https://huggingface.co/datasets/SkillScreen/SkillScreen-Data) · [Code](https://github.com/JerroldH/SkillScreen) · [Release v0.1.0](https://github.com/JerroldH/SkillScreen/releases/tag/v0.1.0) · [Paper](https://arxiv.org/abs/2610.17727) ## Run locally Use Linux, Python 3.12 and a CUDA GPU with BF16 support. The bundled runtime and full context limit are verified on an A100 80GB. ```bash python -m pip install huggingface_hub==1.33.0 hf download SkillScreen/SkillScreen-2B-Package --revision v0.1.0 --local-dir SkillScreen-2B-Package python -m pip install -r SkillScreen-2B-Package/requirements/inference.txt python SkillScreen-2B-Package/scan.py path/to/skill-package --output score.json ``` For a bundled example, replace the input path with `SkillScreen-2B-Package/examples/local-inventory`. The scanner reads files as text and never executes them. Scoring runs locally. `--inspect` lists input files without loading the model. The default is the frozen **99% OOF recall** threshold, approximately **0.288**. `--operating-point native` selects 0.5. Full-precision thresholds are loaded from [deployment_settings.json](deployment_settings.json). Decisions compare raw entailment log-odds, avoiding rounded threshold values. ## Out-of-fold results Three-fold OOF evaluation over **11,831 skills**, including 6,436 malicious and 5,395 benign examples. | Operating point | Score threshold | Recall | Benign FPR | Precision | F1 | | --- | ---: | ---: | ---: | ---: | ---: | | Native 0.5 | 0.500000 | 98.93% | 0.33% | 99.72% | 99.32% | | 99% OOF recall | 0.287766 | 99.01% | 0.35% | 99.70% | 99.35% | **AUPRC (average precision) 0.9990 · AUROC 0.9985.** [Per-skill OOF predictions](evaluation-oof.csv) and [full metrics](evaluation.json) are included. Each OOF score comes from a fold model trained without that skill's content group. Thresholds were selected on these pooled predictions and transferred to the final model trained on all eligible records. These results describe the OOF procedure, not an independent test of the final refit. Recall and FPR can change on new collections. ## Training and input Initialized from [OpenJev 2B](https://huggingface.co/AlexWortega/openjev/tree/a20448012c213128955ca0c693e7c943865cab77/qwen3.5-2b-nli-v5), then trained for one epoch with seed 42, LoRA rank 8 and a trainable NLI classification head. The final weights use all 11,831 eligible records from [SkillScreen-Data at the pinned revision](https://huggingface.co/datasets/SkillScreen/SkillScreen-Data/tree/d59aabbd7e0e8d95b464a2d335f36e948577b832). Labels are inherited from upstream benchmarks. This deployment release is separate from the paper's train-only experiment checkpoints. The fixed hypothesis is “This AI agent skill is malicious.” The score is the entailment softmax output, not a separately calibrated probability. Package text is ordered as SKILL.md, code, configuration, documentation, then bulk text. The report identifies fully retained, partial and omitted files. Inputs above 32,768 tokens are truncated while retaining the hypothesis. A below-threshold score means no model alert, not a safety certification. [Training configuration](training_config.json) · [Adapter](adapter/) · [Release checksums](checksums.json) ## Citation and license Please cite [SkillScreen](https://arxiv.org/abs/2610.17727) for the method and identify this deployment release when reporting results. Machine-readable citation details are in [CITATION.cff](CITATION.cff). Jerrold Huang, Junwei Quan, Saiyue Lyu, Zihan Guo and Yingqiang Ge. *SkillScreen: High-Recall Screening of Malicious Agent Skills with Decision Models.* 2026. Weights, code and configuration are [Apache-2.0](LICENSE) for rights held by the contributors. Documentation and predictions are [CC BY 4.0](LICENSE-CC-BY-4.0). Upstream model and dataset terms remain applicable. See [third-party notices](THIRD_PARTY_NOTICES.md).