---
library_name: transformers
pipeline_tag: text-classification
base_model: AlexWortega/openjev
tags:
- agent-security
- nli
- skilljev
- research
datasets:
- ProtectSkills/MaliciousSkillBench
---

# SkillJev-2B

**SkillJev-Direct v1** scores malicious AI-agent skills from `SKILL.md`, using [OpenJev 2B](https://huggingface.co/AlexWortega/openjev/tree/a20448012c213128955ca0c693e7c943865cab77/qwen3.5-2b-nli-v5), LoRA and a trained classification head. The scoring pipeline returns a calibrated score without generating text.

AI team from University of Toronto, in collaboration with TrendAI.

## Use

The **complete merged model** is at the repository root; the original adapter and head are in [`adapter/`](https://huggingface.co/SkillJev/SkillJev-2B/blob/v1/adapter/README.md). Download version **`v1`**. See [the loading and scoring recipe](https://huggingface.co/SkillJev/SkillJev-2B/blob/v1/USAGE.md) and [`score.py`](https://huggingface.co/SkillJev/SkillJev-2B/blob/v1/score.py).

Premise: `SKILL.md`. Fixed hypothesis:

> This AI agent skill is malicious.

The 8,192-token input retains the Markdown prefix and complete hypothesis. Use [`scoring_config.json`](https://huggingface.co/SkillJev/SkillJev-2B/blob/v1/scoring_config.json) for calibration and the frozen review threshold; native entailment softmax is not the calibrated probability.

## Results

[MaliciousSkillBench](https://huggingface.co/datasets/ProtectSkills/MaliciousSkillBench/tree/d4b42ce5766a6e0359c987cf59c1007cb3795a90), official **Source-Balanced Random** protocol: 6,812 train / 973 validation / 1,950 test.

| Method | Test recall | Benign false-positive rate |
|---|---:|---:|
| **SkillJev-Direct merged v1** | **99.13%** | **5.57%** |
| Word-SVM | 99.40% | 40.98% |
| Untuned OpenJev-2B | 99.47% | 99.78% |

Thresholds were frozen on validation, targeting 99% recall. Merged Direct has 159 fewer false positives and 4 additional misses than Word-SVM. One epoch, one seed; sources are shared across partitions. [Export verification](https://huggingface.co/SkillJev/SkillJev-2B/blob/v1/merge_verification.json) compares the merged model with the selected adapter using unchanged settings.

## Scope

Markdown-only detection does not certify a complete package as safe; performance on other distributions is unverified. OpenJev is labelled MIT; third-party training texts retain their upstream terms. No skill payloads or TrendAI data are hosted here.