sunit12/tiny-aya-global-codemix-refusal-lora

Tiny Aya Global — code-mixed refusal LoRA (circuit-guided, v2)

A 37 MB LoRA for CohereLabs/tiny-aya-global that makes the model refuse harmful requests written in romanized / code-mixed Hindi and Punjabi (and, zero-shot, romanized Bengali, Tamil, Marathi and Urdu) — without runtime hooks, without a system prompt, and with MMLU unchanged.

It is the hook-based guard from Lost in Transliteration (probe-gated refusal steering at layer 14, sunit12/tiny-aya-global-codemix-refusal-guard) distilled into the weights with AlignTune (TRL SFT backend), with LoRA restricted to layers 10–20 — the region our causal patching and CircuitKIT circuit analysis located as where the refusal decision fails for code-mixed input.

Results (held-out prompts, Aya Expanse 8B judge; "usable" = a refusal, not a degenerate loop, in a language the user reads)

	base	hook guard v1.0	this LoRA
usable refusal, held-out harmful, non-English (avg of 4 scripts)	38%	76%	90%
… romanized Hindi / romanized Punjabi	26% / 15%	63% / 63%	91% / 87%
harmless held-out refused (all scripts)	13%	15%	19% (Δ +5 [+1, +10])
unseen languages (romanized bn / mr / ta / ur), usable	15%	57%	78%
chatty filler wrappers: usable / harmless refused	23% / 10%	70% / 28%	86% / 19%
jailbreak wrappers (romanized Hindi), usable — fiction framing	35% — 30%	70% — 30%	75% — 77%
everyday prompts refused (600, EN / HI / PA native)	3%	3%	3.5%
MMLU, 400 q (AuditKIT)	50.5%	50.5%	50.5%

95% CIs are cluster-bootstrap over prompt ids; full tables and logs in sunit12/codemix-safety-logs. Same-data LoRA on all layers reached the same refusal rates but collapsed MMLU (2.5%); without the capability-retention rows (v1) MMLU fell to 30.5% because the model stopped ending its turn after a one-letter answer.

Use

llama.cpp / LM Studio / Ollama (CPU): download codemix-refusal-lora-f16.gguf from this repo and the official CohereLabs/tiny-aya-global-GGUF model, then llama-cli -m tiny-aya-global-q4_k_m.gguf --lora codemix-refusal-lora-f16.gguf.

transformers + peft:

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
tok = AutoTokenizer.from_pretrained("CohereLabs/tiny-aya-global")
model = AutoModelForCausalLM.from_pretrained("CohereLabs/tiny-aya-global", dtype="bfloat16", device_map="auto")
model = PeftModel.from_pretrained(model, "sunit12/tiny-aya-global-codemix-refusal-lora").merge_and_unload()

Training

• Data: sunit12/codemix-refusal-distill-v2, 3,750 rows built only from training prompts: steered-teacher refusals in the user's own script (kept only if judged a refusal, not a loop, in the right script and a readable language by fastText LID; otherwise a short hand-written refusal in that script), self-distilled base-model answers for harmless prompts, story/screenplay/hypothetical/persona-wrapped versions of training prompts, and 700 capability-retention rows (base-model answers to MMLU auxiliary_train questions and everyday prompts).
• AlignTune 0.1.35 create_sft_trainer (TRL backend; CuratorKIT schema gate), LoRA r=16, α=32, dropout 0.05, q/k/v/o/gate/up/down of layers 10–20, lr 2e-4, 2 epochs, bf16, A10G.

Limits

• Harmless false refusals rise by ~4 points on held-out non-English prompts (XSTest-style look-alikes; CI −1 to +9), more on Bengali; +6 points on jailbreak-wrapped harmless prompts.
• Weaker than the hook guard on role-play (73% vs 83%) and translate-then-answer (67% vs 77%) wrappers.
• Many romanized-Hindi refusals reuse one of five hand-written phrasings.
• The translations of the unseen-language test set were machine-written (Claude) and not native-reviewed.
• Inherits the Tiny Aya license (CC-BY-NC 4.0).

Part of Lost in Transliteration (Lexsi Labs hackathon, tracks 2a + 2b).