Jazhyc commited on
Commit
9c4c246
·
verified ·
1 Parent(s): bd86dda

Mark Gleipnir 9B as superseded by Gleipnir 4B v0.2; refresh comparisons

Browse files
Files changed (2) hide show
  1. README.md +19 -7
  2. release_manifest.json +2 -2
README.md CHANGED
@@ -38,6 +38,16 @@ Gleipnir 9B is a research monitor for detecting deception, scheming, and other
38
  control-relevant problematic behavior in visible AI-agent trajectories. It is
39
  a rank-128 LoRA adapter for `Qwen/Qwen3.5-9B`, not a standalone model.
40
 
 
 
 
 
 
 
 
 
 
 
41
  This release is an early research artifact intended for reproducibility and
42
  follow-up work. There is not yet a Gleipnir paper or blog post. It should not be
43
  treated as a production safety system or as evidence that an interaction is
@@ -62,21 +72,23 @@ envelope is recorded in `prompt_contract.json`.
62
 
63
  ## Results
64
 
65
- Both released Gleipnir adapters were evaluated once on the same frozen 6,395-row
66
  strict-OOD suite spanning AgentDojo, CUA-SHADE, IaC, MLE-Sabotage, Rogue Eval,
67
  and SHADE-Arena. The primary metric is the unweighted mean across sources of raw
68
  normalized partial AUROC through 20% false-positive rate.
69
 
70
  | Model | Mean-OOD pAUROC@20 | Mean-OOD AUROC | Pooled pAUROC@20 | Pooled AUROC |
71
  | --- | ---: | ---: | ---: | ---: |
72
- | Gleipnir 4B | 0.7824 | 0.9086 | 0.7015 | 0.9135 |
 
73
  | **Gleipnir 9B** | **0.8212** | **0.9215** | **0.7837** | **0.9355** |
 
74
 
75
- Gleipnir 9B leads Gleipnir 4B by 0.0388 Mean-OOD pAUROC@20 and 0.0130
76
- Mean-OOD AUROC. In the project's dated hosted-inference proxy it costs about
77
- three times as much, so the 4B release remains a strong lower-cost point. These
78
- are single-seed point estimates, and the hosted-cost comparison is not a stable
79
- price quote.
80
 
81
  Adding 13,149 prior deception examples to the 8,688 matched tool-trajectory
82
  examples was macro-neutral relative to the matched-only 9B adapter: Mean-OOD
 
38
  control-relevant problematic behavior in visible AI-agent trajectories. It is
39
  a rank-128 LoRA adapter for `Qwen/Qwen3.5-9B`, not a standalone model.
40
 
41
+ > **Superseded by [Gleipnir 4B](https://huggingface.co/Jazhyc/Gleipnir-4B)
42
+ > (v0.2).** Gleipnir 9B is kept as a research artifact. Gleipnir 4B v0.2
43
+ > performs similarly on the same OOD suite (Mean-OOD pAUROC@20 0.8108 against
44
+ > 0.8212, AUROC 0.9199 against 0.9215) at under a third of the inference
45
+ > cost ($0.26 against $0.87 per 1,000 evaluations in the project's dated
46
+ > hosted-inference proxy). The 1-point pAUROC difference is within the
47
+ > project's 1.8-point seed-to-seed spread. Use Gleipnir 4B for a low-cost
48
+ > monitor, or [Gleipnir 27B](https://huggingface.co/Jazhyc/Gleipnir-27B) for
49
+ > the highest quality.
50
+
51
  This release is an early research artifact intended for reproducibility and
52
  follow-up work. There is not yet a Gleipnir paper or blog post. It should not be
53
  treated as a production safety system or as evidence that an interaction is
 
72
 
73
  ## Results
74
 
75
+ The released Gleipnir adapters were evaluated once on the same frozen 6,395-row
76
  strict-OOD suite spanning AgentDojo, CUA-SHADE, IaC, MLE-Sabotage, Rogue Eval,
77
  and SHADE-Arena. The primary metric is the unweighted mean across sources of raw
78
  normalized partial AUROC through 20% false-positive rate.
79
 
80
  | Model | Mean-OOD pAUROC@20 | Mean-OOD AUROC | Pooled pAUROC@20 | Pooled AUROC |
81
  | --- | ---: | ---: | ---: | ---: |
82
+ | Gleipnir 4B v0.2 | 0.8108 | 0.9199 | 0.7514 | 0.9335 |
83
+ | Gleipnir 4B v0.1 | 0.7824 | 0.9086 | 0.7015 | 0.9135 |
84
  | **Gleipnir 9B** | **0.8212** | **0.9215** | **0.7837** | **0.9355** |
85
+ | Gleipnir 27B (full teacher prompt) | 0.9352 | 0.9790 | 0.9486 | 0.9810 |
86
 
87
+ Gleipnir 9B led the original Gleipnir 4B (v0.1) by 0.0388 Mean-OOD pAUROC@20.
88
+ Against Gleipnir 4B v0.2 (the exact average of three seeds trained on tool
89
+ trajectories only), its lead is 0.0104 pAUROC@20 and 0.0016 AUROC, within
90
+ seed noise, at about 3.3 times the hosted-inference cost. The 9B is a single
91
+ seed. The hosted-cost comparison is not a stable price quote.
92
 
93
  Adding 13,149 prior deception examples to the 8,688 matched tool-trajectory
94
  examples was macro-neutral relative to the matched-only 9B adapter: Mean-OOD
release_manifest.json CHANGED
@@ -8,8 +8,8 @@
8
  "sha256": "c837e257ae9fd8f0372eb30da51cc666600e553f3cdd9da09a47f5b13e08c6f1"
9
  },
10
  "README.md": {
11
- "bytes": 8005,
12
- "sha256": "e66c6744b92bfc978b35bd15d1ca4c6f837d0565d6a506e4f1799a1a43b78e4b"
13
  },
14
  "adapter_config.json": {
15
  "bytes": 1137,
 
8
  "sha256": "c837e257ae9fd8f0372eb30da51cc666600e553f3cdd9da09a47f5b13e08c6f1"
9
  },
10
  "README.md": {
11
+ "bytes": 8805,
12
+ "sha256": "5185ca875dc3e2b5312d1c79e6190845e1aa763ff12f5a0d5f1dc8fac8e58547"
13
  },
14
  "adapter_config.json": {
15
  "bytes": 1137,