SkillJev
/

SkillJev-2B

@@ -10,36 +10,54 @@ datasets:
 - ProtectSkills/MaliciousSkillBench
 ---
 # SkillJev-Direct-2B
 SkillJev-Direct-2B scores malicious AI-agent skills from `SKILL.md`, using [OpenJev 2B](https://huggingface.co/AlexWortega/openjev/tree/a20448012c213128955ca0c693e7c943865cab77/qwen3.5-2b-nli-v5), LoRA and a trained classification head. It returns a calibrated score without generating text.
-A **University of Toronto MScAC AI team research project, developed in collaboration with TrendAI**.
 **Research preview:** training and frozen evaluation are complete; weights are not yet published here.
 ## Results
 [MaliciousSkillBench](https://huggingface.co/datasets/ProtectSkills/MaliciousSkillBench/tree/d4b42ce5766a6e0359c987cf59c1007cb3795a90), official **Source-Balanced Random** protocol: 6,812 train / 973 validation / 1,950 test.
 | Method | Test recall | Benign false-positive rate |
 |---|---:|---:|
 | **SkillJev-Direct-2B** | **99.13%** | **5.57%** |
 | Word-SVM | 99.40% | 40.98% |
 | Untuned OpenJev-2B | 99.47% | 99.78% |
 Thresholds were frozen on validation, targeting 99% recall. Direct has **159 fewer false positives and four additional misses** than Word-SVM. One seed; sources are shared across partitions.
 ## Model input
 Premise: `SKILL.md`. Fixed hypothesis:
 > This AI agent skill is malicious.
 Run `direct-r002`: one epoch, 2.19M trainable parameters, 8,192-token limit. Long inputs retain the Markdown prefix and complete hypothesis. Calibration uses validation only.
 ## Scope and availability
 Markdown-only detection does not certify a complete package as safe; performance on other distributions is unverified.
 Adapter/head release terms are being reviewed. OpenJev is labelled MIT; third-party training texts retain their upstream terms. No skill payloads or TrendAI data are hosted here.

 - ProtectSkills/MaliciousSkillBench
 ---
 # SkillJev-Direct-2B
 SkillJev-Direct-2B scores malicious AI-agent skills from `SKILL.md`, using [OpenJev 2B](https://huggingface.co/AlexWortega/openjev/tree/a20448012c213128955ca0c693e7c943865cab77/qwen3.5-2b-nli-v5), LoRA and a trained classification head. It returns a calibrated score without generating text.
+**Team Lead:** Jerrold Huang
+**Team members:** Junwei Quan, HaoXuan Huang, Shiyi Yang
+University of Toronto MScAC AI team, in collaboration with TrendAI.
 **Research preview:** training and frozen evaluation are complete; weights are not yet published here.
 ## Results
 [MaliciousSkillBench](https://huggingface.co/datasets/ProtectSkills/MaliciousSkillBench/tree/d4b42ce5766a6e0359c987cf59c1007cb3795a90), official **Source-Balanced Random** protocol: 6,812 train / 973 validation / 1,950 test.
 | Method | Test recall | Benign false-positive rate |
 |---|---:|---:|
 | **SkillJev-Direct-2B** | **99.13%** | **5.57%** |
 | Word-SVM | 99.40% | 40.98% |
 | Untuned OpenJev-2B | 99.47% | 99.78% |
 Thresholds were frozen on validation, targeting 99% recall. Direct has **159 fewer false positives and four additional misses** than Word-SVM. One seed; sources are shared across partitions.
 ## Model input
 Premise: `SKILL.md`. Fixed hypothesis:
 > This AI agent skill is malicious.
 Run `direct-r002`: one epoch, 2.19M trainable parameters, 8,192-token limit. Long inputs retain the Markdown prefix and complete hypothesis. Calibration uses validation only.
 ## Scope and availability
 Markdown-only detection does not certify a complete package as safe; performance on other distributions is unverified.
 Adapter/head release terms are being reviewed. OpenJev is labelled MIT; third-party training texts retain their upstream terms. No skill payloads or TrendAI data are hosted here.