MAIN Lab · Qwen2.5 0.5B, Azerbaijani steer (128-number weight edit)

Qwen/Qwen2.5-0.5B-Instruct often answers Azerbaijani prompts in Turkish. In our tests it did so 42% of the time. This model fixes much of that without any training. We changed 128 numbers: one bias vector in layer 20, which is 0.00003% of the model.

Made by MAIN Lab (Master Artificial Intelligence Network). It is a reproduction of the weight-space programming idea from MIT CSAIL ("Sparks of Weight-Space Programming", 2026): read a network's weights and edit them directly.

benchmark

How it was made

  1. We fed the model the same passages in Azerbaijani and in Turkish (parallel FLORES texts from Belebele). The content is identical, so the activation difference is pure language.
  2. In the residual stream, the mean difference forms a clear direction. Near the output it promotes the letters ə, q and x. At layers 18–20 it is almost orthogonal to the Azerbaijani-vs-English direction.
  3. We added a constant shift along that direction at layer 20 (strength 1.5 × the Az–Tr gap). It is written into self_attn.v_proj.bias. Attention weights sum to 1, so this is an exact, always-on residual shift.
  4. Rules:
    • no gradients
    • at most 50 forward-pass "probes" on practice prompts (19 were used)
    • the edit was chosen by a selection rule fixed in advance
    • the test prompts were never seen during selection

The edit itself is in weight_edit.json.

Results (Benchmark #003, 197 held-out Azerbaijani prompts)

Qwen2.5-0.5B + system prompt "answer only in Azerbaijani" This model
Clean Azerbaijani answers 55.3% 57.4% (p = 0.65, not significant) 77.2% (p = 7·10⁻⁷)
Answered in Turkish 41.6% 36.5% 18.3%
Turkish-only words per 100 8.4 6.7 3.9
English prompts answered in English 100% 100% 100%
Fluency, bits/byte under SIBA Meristem (lower is better) 1.332 1.326 1.357

Honest limits:

  • The edit changes which language the model answers in, not its knowledge or reasoning.
  • Reading QA (13.0 → 11.0 F1) and grammar stayed within noise.
  • Fluency is slightly worse.

Use

Use it like the base model; the edit is already applied.

from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("MAIN-Lab/qwen2.5-0.5b-az-steer")
m = AutoModelForCausalLM.from_pretrained("MAIN-Lab/qwen2.5-0.5b-az-steer")

License

Apache-2.0, the same as the base model, Qwen2.5-0.5B-Instruct.

Downloads last month
83
Safetensors
Model size
0.5B params
Tensor type
BF16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for MAIN-Lab/qwen2.5-0.5b-az-steer

Finetuned
(1,194)
this model

Spaces using MAIN-Lab/qwen2.5-0.5b-az-steer 2