Instructions to use SkillScreen/SkillScreen-2B-Markdown with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use SkillScreen/SkillScreen-2B-Markdown with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-classification", model="SkillScreen/SkillScreen-2B-Markdown")# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModelForSequenceClassification processor = AutoProcessor.from_pretrained("SkillScreen/SkillScreen-2B-Markdown") model = AutoModelForSequenceClassification.from_pretrained("SkillScreen/SkillScreen-2B-Markdown", device_map="auto") - Notebooks
- Google Colab
- Kaggle
File size: 4,708 Bytes
b60475f 214e34f b60475f 214e34f b60475f | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 | ---
library_name: transformers
pipeline_tag: text-classification
license: apache-2.0
base_model: AlexWortega/openjev
base_model_relation: finetune
datasets:
- SkillScreen/SkillScreen-Data
tags:
- skillscreen
- agent-security
- decision-model
- nli
- arxiv:2610.17727
---
# SkillScreen-2B-Markdown
**Current deployment release v0.1.0 · 8K context**
SkillScreen-2B-Markdown screens AI-agent skills directly from `SKILL.md`. It returns a maliciousness score and a review decision in one forward pass. Complete merged weights, the original LoRA adapter and frozen operating points are included.
[Package model](https://huggingface.co/SkillScreen/SkillScreen-2B-Package) · [Markdown model](https://huggingface.co/SkillScreen/SkillScreen-2B-Markdown) · [Data](https://huggingface.co/datasets/SkillScreen/SkillScreen-Data) · [Code](https://github.com/JerroldH/SkillScreen) · [Release v0.1.0](https://github.com/JerroldH/SkillScreen/releases/tag/v0.1.0) · [Paper](https://arxiv.org/abs/2610.17727)
## Run locally
Use Linux, Python 3.12 and a CUDA GPU with BF16 support. The bundled runtime and full context limit are verified on an A100 80GB.
```bash
python -m pip install huggingface_hub==1.33.0
hf download SkillScreen/SkillScreen-2B-Markdown --revision v0.1.0 --local-dir SkillScreen-2B-Markdown
python -m pip install -r SkillScreen-2B-Markdown/requirements/inference.txt
python SkillScreen-2B-Markdown/scan.py path/to/SKILL.md --output score.json
```
For a bundled example, replace the input path with `SkillScreen-2B-Markdown/examples/local-inventory/SKILL.md`. The scanner reads files as text and never executes them. Scoring runs locally. `--inspect` lists input files without loading the model.
The default is the **native 0.5** threshold. Use `--operating-point high-recall` for the frozen 99% OOF recall threshold, approximately **0.011642**. `--operating-point native` selects 0.5. Full-precision thresholds are loaded from [deployment_settings.json](deployment_settings.json). Decisions compare raw entailment log-odds, avoiding rounded threshold values.
## Out-of-fold results
Three-fold OOF evaluation over **13,514 skills**, including 7,500 malicious and 6,014 benign examples.
| Operating point | Score threshold | Recall | Benign FPR | Precision | F1 |
| --- | ---: | ---: | ---: | ---: | ---: |
| Native 0.5 | 0.500000 | 96.96% | 0.88% | 99.28% | 98.10% |
| 99% OOF recall | 0.011642 | 99.00% | 15.71% | 88.71% | 93.57% |
**AUPRC (average precision) 0.9966 · AUROC 0.9946.** [Per-skill OOF predictions](evaluation-oof.csv) and [full metrics](evaluation.json) are included.
Each OOF score comes from a fold model trained without that skill's content group. Thresholds were selected on these pooled predictions and transferred to the final model trained on all eligible records. These results describe the OOF procedure, not an independent test of the final refit. Recall and FPR can change on new collections.
## Training and input
Initialized from [OpenJev 2B](https://huggingface.co/AlexWortega/openjev/tree/a20448012c213128955ca0c693e7c943865cab77/qwen3.5-2b-nli-v5), then trained for one epoch with seed 42, LoRA rank 8 and a trainable NLI classification head. The final weights use all 13,514 eligible records from [SkillScreen-Data at the pinned revision](https://huggingface.co/datasets/SkillScreen/SkillScreen-Data/tree/d59aabbd7e0e8d95b464a2d335f36e948577b832). Labels are inherited from upstream benchmarks. This deployment release is separate from the paper's train-only experiment checkpoints.
The fixed hypothesis is “This AI agent skill is malicious.” The score is the entailment softmax output, not a separately calibrated probability. The input is the original Markdown paired with the fixed hypothesis. The report records token truncation. Inputs above 8,192 tokens are truncated while retaining the hypothesis. A below-threshold score means no model alert, not a safety certification.
[Training configuration](training_config.json) · [Adapter](adapter/) · [Release checksums](checksums.json)
## Citation and license
Please cite [SkillScreen](https://arxiv.org/abs/2610.17727) for the method and identify this deployment release when reporting results. Machine-readable citation details are in [CITATION.cff](CITATION.cff).
Jerrold Huang, Junwei Quan, Saiyue Lyu, Zihan Guo and Yingqiang Ge. *SkillScreen: High-Recall Screening of Malicious Agent Skills with Decision Models.* 2026.
Weights, code and configuration are [Apache-2.0](LICENSE) for rights held by the contributors. Documentation and predictions are [CC BY 4.0](LICENSE-CC-BY-4.0). Upstream model and dataset terms remain applicable. See [third-party notices](THIRD_PARTY_NOTICES.md).
|