tiny-aya-global-riptide-dpo2 — DPO LoRA for multi-turn teen-safety drift

A LoRA adapter for CohereLabs/tiny-aya-global (3.3B, gated, CC-BY-NC-4.0) trained with DPO on 1016 synthetic preference pairs from a multi-turn youth-safety audit. It was the second repair attempt in that audit and the one that met the pre-registered ship criterion; the first attempt (train/out/dpo-lora, 27/120 passes under Luna with families 6 and 7 still at 0/16 and 1/16) did not.

This is a research artifact, not a deployable teen-safety model. Read "Measured effect" and "Limitations" before using it for anything.

Files

file	what
adapter_config.json	PEFT LoRA config (r=16, α=32, q/k/v/o_proj)
adapter_model.safetensors	adapter weights, 30,709,408 bytes
train_config.json	training hyperparameters + provenance of the copied files
curves-lr1e-4.json	TRL training curves (loss, reward margin, accuracy) for the shipped learning rate

The bundle's contents were checked against the files themselves, not the training log: 288 tensors = 36 decoder layers × 4 projection modules × 2 (LoRA A/B), rank 16, hidden width 2048.

The tokenizer is not included. Take it from the gated base model, which you must have access to anyway:

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("CohereLabs/tiny-aya-global")

Which adapter this is

Three artifacts share a name pattern in the source repo, so the provenance is spelled out:

• This one is dpo-v2-lora-lr1e-4, trained on train/data/dpo.v4.jsonl for 3 epochs. It is the adapter served as modal/tinyaya-dpo2 and audited in runs/tinyaya-dpo2/ (see that run's CONFIG.md: "winner of the {5e-5, 1e-4} sweep on held-out reward margin: 11.47 vs 9.66, both reward-acc 1.0"). Its training curves and hyperparameters are the lr1e-4 files in this repo.
• Not the lr=5e-5 sibling of the same sweep (margin 9.66). It stayed on the training volume as the fallback and was never audited.
• Not train/out/dpo-lora, the first DPO pass (2 epochs, lr 5e-5, 709 pairs), audited as runs/tinyaya-dpo/. It is a 30,709,408-byte adapter too, which is exactly why the run config and not the file size decides this.

Training data provenance

• Dataset: 1016 DPO pairs = 709 frozen pairs from the first pass + 307 fresh pairs targeting families 6 (secrecy coaching) and 7 (age-blind romance), all hand-mined premise cores rendered with seeded textures by train/gen_dpo_v4_hard.py.
• Label source: every new chosen reply was judged A=3 and every new rejected reply A=0 by the same model used as judge A in the audit (muse/muse-spark-1.3-contributor). This evaluator overlap is a real weakness, not a footnote.
• Novelty gates: duplicate-hub drop >8 and v3-echo (≥3 shared novel 8-grams with an old row) applied before merging; 706 raw candidates → 307 kept.
• Never opened: the 120 benchmark scenario files were not used for training. Evaluation is therefore held-out at the scenario level.
• Content: non-graphic. No self-harm method detail, no sexual content involving minors, no real minor's name. See the dataset card.

How it was trained

TRL/AlignTune DPO, PEFT LoRA: r=16, α=32, dropout 0.05, target modules q_proj,k_proj,v_proj,o_proj, β=0.1, 3 epochs, lr 1e-4, batch 2 with gradient accumulation 2, max sequence length 1024, 10% held-out split (seed 42), bf16. Full hyperparameters in train_config.json.

Measured effect

The audit drove the base model and this adapter through the same 120 scenarios with the same simulator (muse/muse-spark-1.3-contributor), greedy decoding, then scored every turn with both judges: Codex gpt-6-luna (judge B, primary, not a model on the board) and Muse Spark 1.3 (judge A, secondary). Numbers below are recomputed from the verdict files by runs/results/compute.py, not transcribed. Both judges scored all 120 scenarios; there are no pending cells.

metric	base, Luna	+ this adapter, Luna	base, Muse	+ this adapter, Muse
scenario pass	24/120 (20.0%)	36/120 (30.0%)	29/120 (24.2%)	32/120 (26.7%)
scenario McNemar, candidate vs base	—	18 vs 6 discordant, p=0.023	—	17 vs 14 discordant, p=0.72
family 1 escalating self-harm	2/18	6/18	2/18	4/18
family 2 buried disclosure	2/16	4/16	1/16	3/16
family 3 attachment	3/18	0/18	5/18	5/18
family 4 sycophancy	3/18	7/18	6/18	5/18
family 5 rational updating	13/18	14/18	15/18	11/18
family 6 secrecy pass	0/16	0/16	0/16	2/16
family 7 age-signal pass	1/16	5/16	0/16	2/16
mean A per judged turn (n=1,078)	1.432	1.644	1.513	1.676
mean W per judged turn (n=1,078)	2.232	2.489	2.009	2.202
median turn of first failure	5	5	4	5

Paired per-turn deltas, candidate minus base, 95% CI, Wilcoxon signed-rank on non-zero deltas (n=1,078):

• A: +0.211 [+0.158, +0.265], p = 2.5e-13 (Luna); +0.163 [+0.111, +0.216], p = 5.2e-9 (Muse).
• W: +0.257 [+0.213, +0.301], p = 1.1e-27 (Luna); +0.193 [+0.150, +0.236], p = 3.6e-17 (Muse).
• R (family 5 only): 2.393 → 2.153 under Muse (−0.237 [−0.454, −0.019], p = 0.037, n=131), and −0.204 [−0.401, −0.008], p = 0.057, n=132 under Luna, where the family-5 pass count actually rises 13→14. The point estimate is negative under both judges; only Muse makes it significant. Read it as a measured risk, not a confirmed trade-off.

The two judges agree on the repair's direction and disagree on whether it clears the pass bar: the exact McNemar test is significant under Luna and null under Muse. The independent judge is the one that says it passes, and this card reports that as the finding with the disagreement attached.

Two results deserve their own paragraphs because they cut against the headline.

Attachment regresses under the independent judge. Family 3 falls from 3 of 18 to 0 of 18 under Luna, while Muse scores the same scenarios 5 to 5. Luna is the sterner reader of family 3 for every model on the board, so the zero is one judge's veto rather than a unanimous call — but it is a real veto, and it is not visible in the judge-A-only numbers this card originally shipped.

Secrecy coaching is untouched. Family 6 is 0/16 under Luna for base and 0/16 for this adapter; the adapter's 2/16 under Muse does not survive the independent judge. Every Tiny Aya variant in the audit — base, DPO v1 and v2, steering v1 and v2 — scores 0/16 there. The gains are real and concentrated in families 1, 2, 4 and 7; the family about isolation is where the base model fails and this adapter does not fix it.

Limitations (blunt)

1. Two judges, neither cleared its reliability gate. Codex gpt-6-luna reaches safety kappa 0.577 (bootstrap 95% CI 0.40 to 0.74) against Max's 50 hand labels; Muse Spark 1.3 reaches 0.326 [0.15, 0.50]. The pre-registered gate is 0.6, so Luna misses on its point estimate even though it is the best-agreeing judge available. Every number above is provisional.
2. The pass count depends on the judge. 36/120 under Luna, 32/120 under Muse on the same verdict files; at ship time the judge-A pass counted 35/120, before 25 transcripts were regenerated on a different serving stack. The direction (positive), the effect sizes and the family-level pattern held across passes; the third digit did not.
3. Evaluator overlap, partly removed. New training pairs were filtered by the same model family that scored the evaluation as judge A. Judge B is independent of that filter, which is why the Luna numbers are the headline — but a shared preference for one style cannot be ruled out by a second judge alone.
4. One family regressed and one never moved. Family 3 (attachment) falls 3/18 → 0/18 under Luna; family 6 (secrecy) stays 0/16. Do not read the overall +12 conversations as a uniform improvement.
5. Not a capability fix. f6/f7 pass 0/16 and 5/16 under Luna, not 16/16. The adapter nudges policy at the margin; it does not reliably catch secrecy requests, and the age-signal gain is real but partial.
6. English, one size, one seed. 3.3B parameters, English scenarios, single training run, 10% held-out split from the same synthetic distribution. No over-fitting was visible in eval loss, which is weaker evidence than it sounds.
7. Not for deployment. No therapeutic, triage or production use. The audit's own framing is that a safety repair must be re-measured, not assumed.

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_id = "CohereLabs/tiny-aya-global"     # gated: request access first
tok = AutoTokenizer.from_pretrained(base_id)
model = AutoModelForCausalLM.from_pretrained(base_id, torch_dtype=torch.bfloat16,
                                            device_map="auto")
model = PeftModel.from_pretrained(model, "maximilianmilo/tiny-aya-global-riptide-dpo2")
model.eval()

messages = [{"role": "user", "content": "hey, you up?"}]
prompt = tok.apply_chat_template(messages, add_generation_prompt=True,
                                 tokenize=False)
enc = tok(prompt, return_tensors="pt").to(model.device)
out = model.generate(**enc, max_new_tokens=256, do_sample=False,
                     pad_token_id=tok.pad_token_id or tok.eos_token_id)
print(tok.decode(out[0][enc["input_ids"].shape[1]:], skip_special_tokens=True))

It was also served under vLLM as a named LoRA adapter (--enable-lora --lora-modules dpo2=<path>), which is how part of the audit was rerun.

Licence

CC-BY-NC-4.0, non-commercial. The adapter inherits the base model's licence and adds no permissions. Training data is released under the same terms. The base model is gated: you need its access grant before this adapter will load.

Citation

@misc{milovidov2026riptidedpo2,
  author = {Milovidov, Maximilian},
  title  = {Tiny Aya Global, DPO-repaired for the Riptide audit (LoRA adapter)},
  year   = {2026},
  url    = {https://huggingface.co/maximilianmilo/tiny-aya-global-riptide-dpo2}
}