CodeMix Safety Check - models
Collection
Tiny Aya Global and the SafeTune refusal-steering artifact, Lexsi Track 2 โข 2 items โข Updated
An inference-time steering artifact for Cohere Labs' Tiny Aya Global (3.35B), from CodeMix Safety Check (Lexsi Track 2). No weights are changed.
run2/RefusalDirectionTrainer.safetensors: tensor wrapped.direction, used with SafeTune RefusalDirectionModel (mode steer, strength 0.3, all decoder layers). run_fresh_modal.py in the GitHub repo shows the exact loader.1-800-LLMs/tiny-aya-global (ungated copy of CohereLabs/tiny-aya-global).All prompts
| baseline | steered (RD 0.3) | |
|---|---|---|
| Harmful prompts that got harmful help (lower is better) | 4/90 (4.44%) | 0/90 (0.0%) |
| Harmful prompts refused (higher is better) | 81/90 (90.0%) | 87/90 (96.67%) |
| Harmless prompts wrongly refused (lower is better) | 12/140 (8.57%) | 21/140 (15.0%) |
| Harmless prompts answered helpfully (higher is better) | 128/140 (91.43%) | 124/140 (88.57%) |
| Degenerate / looping replies (lower is better) | 10/230 (4.35%) | 8/230 (3.48%) |
English
| baseline | steered (RD 0.3) | |
|---|---|---|
| Harmful prompts that got harmful help (lower is better) | 0/20 (0.0%) | 0/20 (0.0%) |
| Harmful prompts refused (higher is better) | 20/20 (100.0%) | 20/20 (100.0%) |
| Harmless prompts wrongly refused (lower is better) | 0/20 (0.0%) | 0/20 (0.0%) |
| Harmless prompts answered helpfully (higher is better) | 20/20 (100.0%) | 20/20 (100.0%) |
| Degenerate / looping replies (lower is better) | 0/40 (0.0%) | 0/40 (0.0%) |
Hinglish
| baseline | steered (RD 0.3) | |
|---|---|---|
| Harmful prompts that got harmful help (lower is better) | 3/20 (15.0%) | 0/20 (0.0%) |
| Harmful prompts refused (higher is better) | 13/20 (65.0%) | 17/20 (85.0%) |
| Harmless prompts wrongly refused (lower is better) | 0/20 (0.0%) | 0/20 (0.0%) |
| Harmless prompts answered helpfully (higher is better) | 15/20 (75.0%) | 16/20 (80.0%) |
| Degenerate / looping replies (lower is better) | 10/40 (25.0%) | 8/40 (20.0%) |
XSTest
| baseline | steered (RD 0.3) | |
|---|---|---|
| Harmful prompts that got harmful help (lower is better) | 1/50 (2.0%) | 0/50 (0.0%) |
| Harmful prompts refused (higher is better) | 48/50 (96.0%) | 50/50 (100.0%) |
| Harmless prompts wrongly refused (lower is better) | 12/100 (12.0%) | 21/100 (21.0%) |
| Harmless prompts answered helpfully (higher is better) | 93/100 (93.0%) | 88/100 (88.0%) |
| Degenerate / looping replies (lower is better) | 0/150 (0.0%) | 0/150 (0.0%) |
All prompts
| baseline | steered (RD 0.3) | |
|---|---|---|
| Harmful prompts that got harmful help (lower is better) | 9/40 (22.5%) | 2/40 (5.0%) |
| Harmful prompts refused (higher is better) | 23/40 (57.5%) | 32/40 (80.0%) |
| Harmless prompts wrongly refused (lower is better) | 0/40 (0.0%) | 3/40 (7.5%) |
| Harmless prompts answered helpfully (higher is better) | 35/40 (87.5%) | 33/40 (82.5%) |
| Degenerate / looping replies (lower is better) | 12/80 (15.0%) | 15/80 (18.75%) |
hinglish_fresh
| baseline | steered (RD 0.3) | |
|---|---|---|
| Harmful prompts that got harmful help (lower is better) | 9/40 (22.5%) | 2/40 (5.0%) |
| Harmful prompts refused (higher is better) | 23/40 (57.5%) | 32/40 (80.0%) |
| Harmless prompts wrongly refused (lower is better) | 0/40 (0.0%) | 3/40 (7.5%) |
| Harmless prompts answered helpfully (higher is better) | 35/40 (87.5%) | 33/40 (82.5%) |
| Degenerate / looping replies (lower is better) | 12/80 (15.0%) | 15/80 (18.75%) |
The judge flags fewer code-mixed completions as harmful after steering and costs some over-refusal. Known judge false positives, small slices and assistant-authored prompts (not native-speaker validated) are detailed in the full write-up: https://github.com/dhrubasumatary/codemix-safety-check