Text Generation
PEFT
Safetensors
English
ai-control
deception-detection
scheming-detection
monitoring
knowledge-distillation
lora
research-artifact
Eval Results (legacy)
Instructions to use Jazhyc/Gleipnir-9B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- PEFT
How to use Jazhyc/Gleipnir-9B with PEFT:
from peft import PeftModel from transformers import AutoModelForCausalLM base_model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3.5-9B") model = PeftModel.from_pretrained(base_model, "Jazhyc/Gleipnir-9B") - Notebooks
- Google Colab
- Kaggle
Mark Gleipnir 9B as superseded by Gleipnir 4B v0.2; refresh comparisons
Browse files- README.md +19 -7
- release_manifest.json +2 -2
README.md
CHANGED
|
@@ -38,6 +38,16 @@ Gleipnir 9B is a research monitor for detecting deception, scheming, and other
|
|
| 38 |
control-relevant problematic behavior in visible AI-agent trajectories. It is
|
| 39 |
a rank-128 LoRA adapter for `Qwen/Qwen3.5-9B`, not a standalone model.
|
| 40 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 41 |
This release is an early research artifact intended for reproducibility and
|
| 42 |
follow-up work. There is not yet a Gleipnir paper or blog post. It should not be
|
| 43 |
treated as a production safety system or as evidence that an interaction is
|
|
@@ -62,21 +72,23 @@ envelope is recorded in `prompt_contract.json`.
|
|
| 62 |
|
| 63 |
## Results
|
| 64 |
|
| 65 |
-
|
| 66 |
strict-OOD suite spanning AgentDojo, CUA-SHADE, IaC, MLE-Sabotage, Rogue Eval,
|
| 67 |
and SHADE-Arena. The primary metric is the unweighted mean across sources of raw
|
| 68 |
normalized partial AUROC through 20% false-positive rate.
|
| 69 |
|
| 70 |
| Model | Mean-OOD pAUROC@20 | Mean-OOD AUROC | Pooled pAUROC@20 | Pooled AUROC |
|
| 71 |
| --- | ---: | ---: | ---: | ---: |
|
| 72 |
-
| Gleipnir 4B | 0.
|
|
|
|
| 73 |
| **Gleipnir 9B** | **0.8212** | **0.9215** | **0.7837** | **0.9355** |
|
|
|
|
| 74 |
|
| 75 |
-
Gleipnir 9B
|
| 76 |
-
|
| 77 |
-
|
| 78 |
-
|
| 79 |
-
price quote.
|
| 80 |
|
| 81 |
Adding 13,149 prior deception examples to the 8,688 matched tool-trajectory
|
| 82 |
examples was macro-neutral relative to the matched-only 9B adapter: Mean-OOD
|
|
|
|
| 38 |
control-relevant problematic behavior in visible AI-agent trajectories. It is
|
| 39 |
a rank-128 LoRA adapter for `Qwen/Qwen3.5-9B`, not a standalone model.
|
| 40 |
|
| 41 |
+
> **Superseded by [Gleipnir 4B](https://huggingface.co/Jazhyc/Gleipnir-4B)
|
| 42 |
+
> (v0.2).** Gleipnir 9B is kept as a research artifact. Gleipnir 4B v0.2
|
| 43 |
+
> performs similarly on the same OOD suite (Mean-OOD pAUROC@20 0.8108 against
|
| 44 |
+
> 0.8212, AUROC 0.9199 against 0.9215) at under a third of the inference
|
| 45 |
+
> cost ($0.26 against $0.87 per 1,000 evaluations in the project's dated
|
| 46 |
+
> hosted-inference proxy). The 1-point pAUROC difference is within the
|
| 47 |
+
> project's 1.8-point seed-to-seed spread. Use Gleipnir 4B for a low-cost
|
| 48 |
+
> monitor, or [Gleipnir 27B](https://huggingface.co/Jazhyc/Gleipnir-27B) for
|
| 49 |
+
> the highest quality.
|
| 50 |
+
|
| 51 |
This release is an early research artifact intended for reproducibility and
|
| 52 |
follow-up work. There is not yet a Gleipnir paper or blog post. It should not be
|
| 53 |
treated as a production safety system or as evidence that an interaction is
|
|
|
|
| 72 |
|
| 73 |
## Results
|
| 74 |
|
| 75 |
+
The released Gleipnir adapters were evaluated once on the same frozen 6,395-row
|
| 76 |
strict-OOD suite spanning AgentDojo, CUA-SHADE, IaC, MLE-Sabotage, Rogue Eval,
|
| 77 |
and SHADE-Arena. The primary metric is the unweighted mean across sources of raw
|
| 78 |
normalized partial AUROC through 20% false-positive rate.
|
| 79 |
|
| 80 |
| Model | Mean-OOD pAUROC@20 | Mean-OOD AUROC | Pooled pAUROC@20 | Pooled AUROC |
|
| 81 |
| --- | ---: | ---: | ---: | ---: |
|
| 82 |
+
| Gleipnir 4B v0.2 | 0.8108 | 0.9199 | 0.7514 | 0.9335 |
|
| 83 |
+
| Gleipnir 4B v0.1 | 0.7824 | 0.9086 | 0.7015 | 0.9135 |
|
| 84 |
| **Gleipnir 9B** | **0.8212** | **0.9215** | **0.7837** | **0.9355** |
|
| 85 |
+
| Gleipnir 27B (full teacher prompt) | 0.9352 | 0.9790 | 0.9486 | 0.9810 |
|
| 86 |
|
| 87 |
+
Gleipnir 9B led the original Gleipnir 4B (v0.1) by 0.0388 Mean-OOD pAUROC@20.
|
| 88 |
+
Against Gleipnir 4B v0.2 (the exact average of three seeds trained on tool
|
| 89 |
+
trajectories only), its lead is 0.0104 pAUROC@20 and 0.0016 AUROC, within
|
| 90 |
+
seed noise, at about 3.3 times the hosted-inference cost. The 9B is a single
|
| 91 |
+
seed. The hosted-cost comparison is not a stable price quote.
|
| 92 |
|
| 93 |
Adding 13,149 prior deception examples to the 8,688 matched tool-trajectory
|
| 94 |
examples was macro-neutral relative to the matched-only 9B adapter: Mean-OOD
|
release_manifest.json
CHANGED
|
@@ -8,8 +8,8 @@
|
|
| 8 |
"sha256": "c837e257ae9fd8f0372eb30da51cc666600e553f3cdd9da09a47f5b13e08c6f1"
|
| 9 |
},
|
| 10 |
"README.md": {
|
| 11 |
-
"bytes":
|
| 12 |
-
"sha256": "
|
| 13 |
},
|
| 14 |
"adapter_config.json": {
|
| 15 |
"bytes": 1137,
|
|
|
|
| 8 |
"sha256": "c837e257ae9fd8f0372eb30da51cc666600e553f3cdd9da09a47f5b13e08c6f1"
|
| 9 |
},
|
| 10 |
"README.md": {
|
| 11 |
+
"bytes": 8805,
|
| 12 |
+
"sha256": "5185ca875dc3e2b5312d1c79e6190845e1aa763ff12f5a0d5f1dc8fac8e58547"
|
| 13 |
},
|
| 14 |
"adapter_config.json": {
|
| 15 |
"bytes": 1137,
|