Tiny Aya Global + CodeMix refusal steer

An inference-time steering artifact for Cohere Labs' Tiny Aya Global (3.35B), from CodeMix Safety Check (Lexsi Track 2). No weights are changed.

  • run2/RefusalDirectionTrainer.safetensors: tensor wrapped.direction, used with SafeTune RefusalDirectionModel (mode steer, strength 0.3, all decoder layers). run_fresh_modal.py in the GitHub repo shows the exact loader.
  • Base weights: 1-800-LLMs/tiny-aya-global (ungated copy of CohereLabs/tiny-aya-global).

Results (blinded Qwen2.5-32B four-label judge; estimates, not certified safety)

Held-out exploratory set (230)

All prompts

baseline steered (RD 0.3)
Harmful prompts that got harmful help (lower is better) 4/90 (4.44%) 0/90 (0.0%)
Harmful prompts refused (higher is better) 81/90 (90.0%) 87/90 (96.67%)
Harmless prompts wrongly refused (lower is better) 12/140 (8.57%) 21/140 (15.0%)
Harmless prompts answered helpfully (higher is better) 128/140 (91.43%) 124/140 (88.57%)
Degenerate / looping replies (lower is better) 10/230 (4.35%) 8/230 (3.48%)

English

baseline steered (RD 0.3)
Harmful prompts that got harmful help (lower is better) 0/20 (0.0%) 0/20 (0.0%)
Harmful prompts refused (higher is better) 20/20 (100.0%) 20/20 (100.0%)
Harmless prompts wrongly refused (lower is better) 0/20 (0.0%) 0/20 (0.0%)
Harmless prompts answered helpfully (higher is better) 20/20 (100.0%) 20/20 (100.0%)
Degenerate / looping replies (lower is better) 0/40 (0.0%) 0/40 (0.0%)

Hinglish

baseline steered (RD 0.3)
Harmful prompts that got harmful help (lower is better) 3/20 (15.0%) 0/20 (0.0%)
Harmful prompts refused (higher is better) 13/20 (65.0%) 17/20 (85.0%)
Harmless prompts wrongly refused (lower is better) 0/20 (0.0%) 0/20 (0.0%)
Harmless prompts answered helpfully (higher is better) 15/20 (75.0%) 16/20 (80.0%)
Degenerate / looping replies (lower is better) 10/40 (25.0%) 8/40 (20.0%)

XSTest

baseline steered (RD 0.3)
Harmful prompts that got harmful help (lower is better) 1/50 (2.0%) 0/50 (0.0%)
Harmful prompts refused (higher is better) 48/50 (96.0%) 50/50 (100.0%)
Harmless prompts wrongly refused (lower is better) 12/100 (12.0%) 21/100 (21.0%)
Harmless prompts answered helpfully (higher is better) 93/100 (93.0%) 88/100 (88.0%)
Degenerate / looping replies (lower is better) 0/150 (0.0%) 0/150 (0.0%)

Fresh Hinglish validation (80, frozen before generation)

All prompts

baseline steered (RD 0.3)
Harmful prompts that got harmful help (lower is better) 9/40 (22.5%) 2/40 (5.0%)
Harmful prompts refused (higher is better) 23/40 (57.5%) 32/40 (80.0%)
Harmless prompts wrongly refused (lower is better) 0/40 (0.0%) 3/40 (7.5%)
Harmless prompts answered helpfully (higher is better) 35/40 (87.5%) 33/40 (82.5%)
Degenerate / looping replies (lower is better) 12/80 (15.0%) 15/80 (18.75%)

hinglish_fresh

baseline steered (RD 0.3)
Harmful prompts that got harmful help (lower is better) 9/40 (22.5%) 2/40 (5.0%)
Harmful prompts refused (higher is better) 23/40 (57.5%) 32/40 (80.0%)
Harmless prompts wrongly refused (lower is better) 0/40 (0.0%) 3/40 (7.5%)
Harmless prompts answered helpfully (higher is better) 35/40 (87.5%) 33/40 (82.5%)
Degenerate / looping replies (lower is better) 12/80 (15.0%) 15/80 (18.75%)

The judge flags fewer code-mixed completions as harmful after steering and costs some over-refusal. Known judge false positives, small slices and assistant-authored prompts (not native-speaker validated) are detailed in the full write-up: https://github.com/dhrubasumatary/codemix-safety-check

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for dhrubasumatary/tiny-aya-global-codemix-steer

Finetuned
(171)
this model

Collection including dhrubasumatary/tiny-aya-global-codemix-steer