| --- |
| library_name: transformers |
| pipeline_tag: text-classification |
| base_model: AlexWortega/openjev |
| tags: |
| - agent-security |
| - nli |
| - skilljev |
| - research |
| datasets: |
| - ProtectSkills/MaliciousSkillBench |
| --- |
| |
| # SkillJev-2B |
|
|
| **SkillJev-Direct v1** scores malicious AI-agent skills from `SKILL.md`, using [OpenJev 2B](https://huggingface.co/AlexWortega/openjev/tree/a20448012c213128955ca0c693e7c943865cab77/qwen3.5-2b-nli-v5), LoRA and a trained classification head. The scoring pipeline returns a calibrated score without generating text. |
|
|
| AI team from University of Toronto, in collaboration with TrendAI. |
|
|
| ## Use |
|
|
| The **complete merged model** is at the repository root; the original adapter and head are in [`adapter/`](https://huggingface.co/SkillJev/SkillJev-2B/blob/v1/adapter/README.md). Download version **`v1`**. See [the loading and scoring recipe](https://huggingface.co/SkillJev/SkillJev-2B/blob/v1/USAGE.md) and [`score.py`](https://huggingface.co/SkillJev/SkillJev-2B/blob/v1/score.py). |
|
|
| Premise: `SKILL.md`. Fixed hypothesis: |
|
|
| > This AI agent skill is malicious. |
|
|
| The 8,192-token input retains the Markdown prefix and complete hypothesis. Use [`scoring_config.json`](https://huggingface.co/SkillJev/SkillJev-2B/blob/v1/scoring_config.json) for calibration and the frozen review threshold; native entailment softmax is not the calibrated probability. |
|
|
| ## Results |
|
|
| [MaliciousSkillBench](https://huggingface.co/datasets/ProtectSkills/MaliciousSkillBench/tree/d4b42ce5766a6e0359c987cf59c1007cb3795a90), official **Source-Balanced Random** protocol: 6,812 train / 973 validation / 1,950 test. |
|
|
| | Method | Test recall | Benign false-positive rate | |
| |---|---:|---:| |
| | **SkillJev-Direct merged v1** | **99.13%** | **5.57%** | |
| | Word-SVM | 99.40% | 40.98% | |
| | Untuned OpenJev-2B | 99.47% | 99.78% | |
|
|
| Thresholds were frozen on validation, targeting 99% recall. Merged Direct has 159 fewer false positives and 4 additional misses than Word-SVM. One epoch, one seed; sources are shared across partitions. [Export verification](https://huggingface.co/SkillJev/SkillJev-2B/blob/v1/merge_verification.json) compares the merged model with the selected adapter using unchanged settings. |
|
|
| ## Scope |
|
|
| Markdown-only detection does not certify a complete package as safe; performance on other distributions is unverified. OpenJev is labelled MIT; third-party training texts retain their upstream terms. No skill payloads or TrendAI data are hosted here. |
|
|
