license: cc-by-nc-4.0
base_model: CohereLabs/tiny-aya-fire
tags:
- safety
- steering
- refusal
- multilingual
- code-mixing
- hindi
- punjabi
- romanized
- tiny-aya
language:
- en
- hi
- pa
Tiny Aya Fire — CodeMix Refusal Guard
The Tiny Aya Global guard re-fit for
CohereLabs/tiny-aya-fire with the identical recipe — a transfer test of the method from
Lost in Transliteration. No weights change: a logistic probe on the last prompt token at layer 13
(threshold 0.6, chosen on a validation split of training data) gates the addition of the English diff-in-means
refusal vector at layer 14 (α = 1).
Results (held-out; judge: Aya Expanse 8B; 95% cluster-bootstrap CIs)
| Tiny Aya Fire | Plain | + guard |
|---|---|---|
| Refused, harmful English / romanized Hindi / romanized Punjabi | 85% / 35% / 30% | 96% / 89% / 78% |
| Usable refusal, non-English harmful (184) | 47% [39, 54] | 76% [69, 82] (+29 [+20, +38]) |
| Complied, non-English harmful | 30% | 7% |
| Unseen India-specific harms, usable refusal (80) | 60% | 84% (+24 [+11, +37]) |
| Harmless-but-scary refused (/240) | 35 | 38 |
| Gate fires: held-out harmful / harmless / everyday (600) / romanized everyday | — | 92% / 1.7% / 0.3% / 2.7% |
| MMLU (AuditKIT, 400 q) | 51.0% | 51.0% (gate fired 0 / 400) |
Fire shows the same romanization gap as Global, and the same fix closes most of it. Global's gate applied to Fire without refitting still catches 90% of held-out harmful prompts (66% of unseen categories), so most of the gate transfers between the two models; the per-model refit is better on new categories (77%).
Use
Same guard.py as the Global guard:
guard = guard_mod.CodeMixRefusalGuard.from_pretrained("sunit12/tiny-aya-fire-codemix-refusal-guard").attach(model) # model = tiny-aya-fire
Files: gate_v3.npz (Fire gate), refusal_vectors.npz (Fire diff-in-means vectors, all 36 layers), guard.py.
Code, data and logs: https://github.com/SunitSinghDhaliwal/lost-in-transliteration ·
sunit12/codemix-safety-bench · sunit12/codemix-safety-logs.
Limitations
Same as the Global guard: it restores the refusal decision, not comprehension of romanized text (some restored romanized-Punjabi refusals are in the wrong language); chatty wrappers can trigger the gate on harmless prompts; an inference-time hook is removable by anyone with the weights. SafeTune CAST was not re-run on Fire.