CAA vectors, layer 16 @ alpha 1.5 (negative result)

README.md

@@ -0,0 +1,177 @@
+---
+license: cc-by-nc-4.0
+language:
+- en
+library_name: safetune
+tags:
+- steering
+- caa
+- activation-steering
+- safety
+- youth-safety
+- riptide
+base_model: CohereLabs/tiny-aya-global
+---
+
+# riptide steer-v2 — CAA steering vector (layer 16, α 1.5)
+
+Contrastive activation addition (CAA) vectors for
+[`CohereLabs/tiny-aya-global`](https://huggingface.co/CohereLabs/tiny-aya-global)
+(3.3B, gated, CC-BY-NC-4.0), produced by the Riptide multi-turn
+youth-safety audit as its second steering attempt.
+
+**Do not read this as a safety fix.** The audit measured it as a
+regression: no family 6 or 7 gain (0/16 and 1/16 under the primary judge,
+unchanged from base), a small but significant drop in the safety axis, and
+19/120 scenario passes against the base model's 24/120. It is published so
+the negative result can be reproduced, not deployed.
+
+## What is in here
+
+| file | what |
+|---|---|
+| `vectors.pt` | `dict[int, Tensor]`, 5 vectors, 2048-dim `float32` each — layers 16, 20, 22, 24, 28 |
+| `config.json` | selected layer/α, the sweep grid, extraction settings |
+| `sweep.json` | all 20 layer × α cells: held-out margin, benign perplexity, selection rule |
+
+## Where it applies
+
+- **Model:** `CohereLabs/tiny-aya-global`, 36 decoder layers.
+- **Layer:** index **16** (0-based index into the model's decoder-layer
+  list — the same indexing the `vectors.pt` dict keys use), of 36.
+- **Hook:** a forward hook on that decoder layer's residual-stream output,
+  installed by SafeTune's `CAAModel`. The steering vector is added to the
+  activations during generation; no weights are modified, so removing the
+  hook restores the base model exactly.
+- **Strength:** α = **1.5** (`strength` in `CAAModel`), i.e. the vector is
+  scaled 1.5× before it is added.
+- **Only layer 16 is used.** The other four are the losing sweep cells,
+  kept so the sweep can be re-scored without re-extracting.
+
+## How to load it
+
+This is the exact code path used in the audit
+(`infra/hf_serve_steer.py`, `Steer2Chat` in `infra/modal_repair_v2.py`):
+
+```python
+import json
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+from safetune.interventions.steer.caa import CAAModel   # pip: safetune
+
+base_id = "CohereLabs/tiny-aya-global"                  # gated: request access first
+tok = AutoTokenizer.from_pretrained(base_id, trust_remote_code=False)
+model = AutoModelForCausalLM.from_pretrained(
+    base_id, torch_dtype=torch.bfloat16, device_map="auto",
+    trust_remote_code=False)
+model.eval()
+
+cfg = json.load(open("config.json"))
+vecs = torch.load("vectors.pt", map_location="cpu", weights_only=True)
+layer, alpha = int(cfg["layer"]), float(cfg["alpha"])
+dev = next(model.parameters()).device
+CAAModel(model, {layer: vecs[layer].to(dev)}, strength=alpha).install()
+
+messages = [{"role": "user", "content": "hey, you up?"}]
+prompt = tok.apply_chat_template(messages, add_generation_prompt=True,
+                                 tokenize=False)
+enc = tok(prompt, return_tensors="pt").to(dev)
+out = model.generate(**enc, max_new_tokens=256, do_sample=False,
+                     pad_token_id=tok.pad_token_id or tok.eos_token_id)
+print(tok.decode(out[0][enc["input_ids"].shape[1]:], skip_special_tokens=True))
+```
+
+`vectors.pt` is a plain `torch.save` dict, so `torch.load(..., weights_only=True)`
+is enough — no SafeTune import is needed to *inspect* the vectors:
+
+```python
+print({k: tuple(v.shape) for k, v in vecs.items()})
+# {16: (2048,), 20: (2048,), 22: (2048,), 24: (2048,), 28: (2048,)}
+```
+
+## How it was extracted
+
+- **Pairs:** 115 extraction pairs, drawn from a 135-pair contrastive set
+  (prompt + chosen `positive` + failing `negative` reply), 20 held out for
+  the sweep. Seed 7.
+- **Pooling:** last-token activations, no normalization.
+- **Selection:** grid over layers {16, 20, 22, 24, 28} × α {0.5, 0.8, 1.0,
+  1.5}; pick the cell with the largest held-out margin gain whose benign
+  perplexity rise stays under 5%. 19 of 20 cells were feasible.
+- **Selected cell:** layer 16, α 1.5 — held-out margin **−187.7 → −184.25**
+  (+3.45), benign perplexity **29.22 → 29.84** (+2.1%).
+- Full table in `sweep.json` (`base_margin`, `results[]`, `best`).
+
+Interpretation from the audit: the largest gain available from this
+extraction was a nudge. The margin stayed deeply negative, so the
+underlying preference for the failing continuation never flipped.
+
+## Measured effect (the negative result)
+
+Same 120 scenarios, same simulator, greedy decoding, both judges
+(Codex `gpt-6-luna` primary, Muse Spark 1.3 secondary). Both scored all
+120 scenarios; there are no pending cells.
+
+| metric | base, Luna | + this vector, Luna | base, Muse | + this vector, Muse |
+|---|---|---|---|---|
+| scenario pass | **24/120** | **19/120** | **29/120** | **20/120** |
+| scenario McNemar, candidate vs base | — | 6 vs 11 discordant, p=0.33 | — | 4 vs 13 discordant, p=0.049 |
+| family 6 secrecy pass | 0/16 | 0/16 | 0/16 | 0/16 |
+| family 7 age-signal pass | 1/16 | 1/16 | 0/16 | 0/16 |
+| family 3 attachment | 3/18 | 2/18 | 5/18 | 4/18 |
+| mean A per judged turn | 1.432 | 1.380 | 1.513 | 1.440 |
+| mean W per judged turn | 2.232 | 2.118 | 2.009 | 1.976 |
+| median turn of first failure | 5 | 4 | 4 | 4 |
+
+Paired per-turn deltas, candidate minus base (n=1,078):
+
+- **A: −0.052 [−0.098, −0.006], p = 0.036 (Luna); −0.073 [−0.120,
+  −0.026], p = 0.002 (Muse).**
+- **W: −0.114 [−0.156, −0.072], p = 1.9e-7 (Luna); −0.033 [−0.072,
+  +0.005], p = 0.080 (Muse).**
+
+Steering harder than the first attempt made safety slightly worse, not
+better, and the two judges agree on the direction at the turn level. They
+disagree on the scenario level: the loss in passes is marginal under Muse
+(p=0.049) and not significant under Luna (p=0.33), so the honest summary
+is a reliable per-turn degradation that may or may not cost whole
+conversations. Against the DPO adapter the ordering is unambiguous under
+both judges: 19 versus 36 passes under Luna, 5 vs 22 discordant,
+p=0.0015, mean A −0.264. The audit closed the steering line here; the DPO
+adapter was shipped instead.
+
+## Limitations
+
+1. **Two judges, neither gate-cleared.** Codex `gpt-6-luna` reaches safety
+   kappa 0.577 (bootstrap 95% CI 0.40 to 0.74) on Max's 50 hand labels;
+   Muse Spark 1.3 reaches 0.326 [0.15, 0.50]. The pre-registered gate is
+   0.6. The regression's direction is consistent across both judges; its
+   size is not.
+2. **One extraction, one seed, one model family.** CAA vectors are
+   model-specific: these weights mean nothing for another checkpoint, even
+   a same-size one.
+3. **The pass count moved between judge passes:** 21/120 at ship time,
+   20/120 under Muse after 25 transcripts were regenerated on a different
+   serving stack and re-judged, and 19/120 under Luna. All three are
+   regressions against base; the exact size is judge-pass dependent.
+4. **The vector nudges preferences, it does not add a policy.** It cannot
+   teach a behaviour the base model never shows; here it failed to move
+   families 6 and 7 at all under either judge.
+5. **Not for deployment.** No therapeutic, triage or production use.
+
+## Licence
+
+**CC-BY-NC-4.0, non-commercial.** The vectors are derived from the gated
+base model's activations and inherit its licence. Extraction pairs come
+from the Riptide training set, released under the same terms.
+
+## Citation
+
+```
+@misc{milovidov2026riptidesteer2,
+  author = {Milovidov, Maximilian},
+  title  = {Tiny Aya Global, CAA steering vectors from the Riptide audit (negative result)},
+  year   = {2026},
+  url    = {https://huggingface.co/maximilianmilo/tiny-aya-global-riptide-steer2}
+}
+```

config.json

@@ -0,0 +1,25 @@
+{
+  "layer": 16,
+  "alpha": 1.5,
+  "grid_layers": [
+    16,
+    20,
+    22,
+    24,
+    28
+  ],
+  "grid_alphas": [
+    0.5,
+    0.8,
+    1.0,
+    1.5
+  ],
+  "n_layers": 36,
+  "pool_method": "last_token",
+  "normalize": false,
+  "held_out": 20,
+  "extract_on": 115,
+  "benign_controls": 20,
+  "seed": 7,
+  "select_rule": "max delta_margin with rel_ppl<0.05"
+}

sweep.json

@@ -0,0 +1,259 @@
+{
+ "base_margin": -187.7,
+ "base_benign_ppl": 29.21724602478943,
+ "results": [
+  {
+   "layer": 16,
+   "alpha": 0.5,
+   "n": 20,
+   "base_margin": -187.7,
+   "margin": -186.15,
+   "delta_margin": 1.55,
+   "base_benign_ppl": 29.2172,
+   "benign_ppl": 29.161,
+   "benign_tokens": 2076,
+   "rel_ppl": -0.0019
+  },
+  {
+   "layer": 16,
+   "alpha": 0.8,
+   "n": 20,
+   "base_margin": -187.7,
+   "margin": -184.875,
+   "delta_margin": 2.825,
+   "base_benign_ppl": 29.2172,
+   "benign_ppl": 29.1891,
+   "benign_tokens": 2076,
+   "rel_ppl": -0.001
+  },
+  {
+   "layer": 16,
+   "alpha": 1.0,
+   "n": 20,
+   "base_margin": -187.7,
+   "margin": -184.5,
+   "delta_margin": 3.2,
+   "base_benign_ppl": 29.2172,
+   "benign_ppl": 29.4149,
+   "benign_tokens": 2076,
+   "rel_ppl": 0.0068
+  },
+  {
+   "layer": 16,
+   "alpha": 1.5,
+   "n": 20,
+   "base_margin": -187.7,
+   "margin": -184.25,
+   "delta_margin": 3.45,
+   "base_benign_ppl": 29.2172,
+   "benign_ppl": 29.8431,
+   "benign_tokens": 2076,
+   "rel_ppl": 0.0214
+  },
+  {
+   "layer": 20,
+   "alpha": 0.5,
+   "n": 20,
+   "base_margin": -187.7,
+   "margin": -185.95,
+   "delta_margin": 1.75,
+   "base_benign_ppl": 29.2172,
+   "benign_ppl": 28.4123,
+   "benign_tokens": 2076,
+   "rel_ppl": -0.0276
+  },
+  {
+   "layer": 20,
+   "alpha": 0.8,
+   "n": 20,
+   "base_margin": -187.7,
+   "margin": -185.65,
+   "delta_margin": 2.05,
+   "base_benign_ppl": 29.2172,
+   "benign_ppl": 28.2213,
+   "benign_tokens": 2076,
+   "rel_ppl": -0.0341
+  },
+  {
+   "layer": 20,
+   "alpha": 1.0,
+   "n": 20,
+   "base_margin": -187.7,
+   "margin": -185.125,
+   "delta_margin": 2.575,
+   "base_benign_ppl": 29.2172,
+   "benign_ppl": 28.1399,
+   "benign_tokens": 2076,
+   "rel_ppl": -0.0369
+  },
+  {
+   "layer": 20,
+   "alpha": 1.5,
+   "n": 20,
+   "base_margin": -187.7,
+   "margin": -184.7,
+   "delta_margin": 3.0,
+   "base_benign_ppl": 29.2172,
+   "benign_ppl": 28.3849,
+   "benign_tokens": 2076,
+   "rel_ppl": -0.0285
+  },
+  {
+   "layer": 22,
+   "alpha": 0.5,
+   "n": 20,
+   "base_margin": -187.7,
+   "margin": -186.325,
+   "delta_margin": 1.375,
+   "base_benign_ppl": 29.2172,
+   "benign_ppl": 28.5495,
+   "benign_tokens": 2076,
+   "rel_ppl": -0.0229
+  },
+  {
+   "layer": 22,
+   "alpha": 0.8,
+   "n": 20,
+   "base_margin": -187.7,
+   "margin": -186.3,
+   "delta_margin": 1.4,
+   "base_benign_ppl": 29.2172,
+   "benign_ppl": 28.3303,
+   "benign_tokens": 2076,
+   "rel_ppl": -0.0304
+  },
+  {
+   "layer": 22,
+   "alpha": 1.0,
+   "n": 20,
+   "base_margin": -187.7,
+   "margin": -186.125,
+   "delta_margin": 1.575,
+   "base_benign_ppl": 29.2172,
+   "benign_ppl": 28.4123,
+   "benign_tokens": 2076,
+   "rel_ppl": -0.0276
+  },
+  {
+   "layer": 22,
+   "alpha": 1.5,
+   "n": 20,
+   "base_margin": -187.7,
+   "margin": -187.15,
+   "delta_margin": 0.55,
+   "base_benign_ppl": 29.2172,
+   "benign_ppl": 28.8814,
+   "benign_tokens": 2076,
+   "rel_ppl": -0.0115
+  },
+  {
+   "layer": 24,
+   "alpha": 0.5,
+   "n": 20,
+   "base_margin": -187.7,
+   "margin": -187.45,
+   "delta_margin": 0.25,
+   "base_benign_ppl": 29.2172,
+   "benign_ppl": 28.7981,
+   "benign_tokens": 2076,
+   "rel_ppl": -0.0143
+  },
+  {
+   "layer": 24,
+   "alpha": 0.8,
+   "n": 20,
+   "base_margin": -187.7,
+   "margin": -187.65,
+   "delta_margin": 0.05,
+   "base_benign_ppl": 29.2172,
+   "benign_ppl": 28.6873,
+   "benign_tokens": 2076,
+   "rel_ppl": -0.0181
+  },
+  {
+   "layer": 24,
+   "alpha": 1.0,
+   "n": 20,
+   "base_margin": -187.7,
+   "margin": -187.825,
+   "delta_margin": -0.125,
+   "base_benign_ppl": 29.2172,
+   "benign_ppl": 28.7426,
+   "benign_tokens": 2076,
+   "rel_ppl": -0.0162
+  },
+  {
+   "layer": 24,
+   "alpha": 1.5,
+   "n": 20,
+   "base_margin": -187.7,
+   "margin": -189.175,
+   "delta_margin": -1.475,
+   "base_benign_ppl": 29.2172,
+   "benign_ppl": 29.2454,
+   "benign_tokens": 2076,
+   "rel_ppl": 0.001
+  },
+  {
+   "layer": 28,
+   "alpha": 0.5,
+   "n": 20,
+   "base_margin": -187.7,
+   "margin": -188.225,
+   "delta_margin": -0.525,
+   "base_benign_ppl": 29.2172,
+   "benign_ppl": 29.3583,
+   "benign_tokens": 2076,
+   "rel_ppl": 0.0048
+  },
+  {
+   "layer": 28,
+   "alpha": 0.8,
+   "n": 20,
+   "base_margin": -187.7,
+   "margin": -189.225,
+   "delta_margin": -1.525,
+   "base_benign_ppl": 29.2172,
+   "benign_ppl": 29.5855,
+   "benign_tokens": 2076,
+   "rel_ppl": 0.0126
+  },
+  {
+   "layer": 28,
+   "alpha": 1.0,
+   "n": 20,
+   "base_margin": -187.7,
+   "margin": -190.15,
+   "delta_margin": -2.45,
+   "base_benign_ppl": 29.2172,
+   "benign_ppl": 29.9583,
+   "benign_tokens": 2076,
+   "rel_ppl": 0.0254
+  },
+  {
+   "layer": 28,
+   "alpha": 1.5,
+   "n": 20,
+   "base_margin": -187.7,
+   "margin": -192.35,
+   "delta_margin": -4.65,
+   "base_benign_ppl": 29.2172,
+   "benign_ppl": 31.2555,
+   "benign_tokens": 2076,
+   "rel_ppl": 0.0698
+  }
+ ],
+ "best": {
+  "layer": 16,
+  "alpha": 1.5,
+  "n": 20,
+  "base_margin": -187.7,
+  "margin": -184.25,
+  "delta_margin": 3.45,
+  "base_benign_ppl": 29.2172,
+  "benign_ppl": 29.8431,
+  "benign_tokens": 2076,
+  "rel_ppl": 0.0214
+ },
+ "feasible_count": 19
+}

vectors.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5801745db057009de9b13215c3f75ec2a2475830cb5c489bc1f5b116f0264e56
+size 43225