Qwen3.5-0.8B (e2e pilot) โ€” Soft MoE expert ยซPython language & PEPsยป

EN

UPDATE: Benchmark results & v2 rebuild

This first build was re-tested in Ivanna Lab (self-hosted LLM model lab, GitHub) with its benchmarks module: code2text_python (CodeXGLUE) smoothed BLEU-4 = 1.0289 (150 samples). A v2 rebuild (alpha=0.1, position=last, RAG-sampled domain prompts) scored 1.0534 โ€” see the successor repo Qwen3.5-0.8B-Python-SoftMoE-v2.

Benchmark chart

A soft MoE domain expert derived from a dense model without any training: the domain direction dฬ‚ is extracted via activation scans (mean hidden-state difference between domain and contrast prompts) and baked into the weights with a rank-one steer patch on output matrices of selected layers. An embedding-space router (cosine margin) picks this variant instead of the base model. Patch mode: steer, alpha = 0.2, layers: [15, 16, 17, 18, 20, 21, 22, 23]. Scan gate AUC = 1.0.

Created with Ivanna Lab โ€” self-hosted LLM model lab.

Notes

Domain: Python language and PEPs. Scan AUC 1.0. Full eval gate included.

RU

ะœัะณะบะธะน MoE-ัะบัะฟะตั€ั‚, ะฟะพะปัƒั‡ะตะฝะฝั‹ะน ะธะท ะฟะปะพั‚ะฝะพะน ะผะพะดะตะปะธ ะฑะตะท ะดะพะพะฑัƒั‡ะตะฝะธั (weight-space steering): ะฝะฐะฟั€ะฐะฒะปะตะฝะธะต ะดะพะผะตะฝะฐ dฬ‚ ะฒั‹ะดะตะปะตะฝะพ ะฐะบั‚ะธะฒะฐั†ะธะพะฝะฝั‹ะผ ัะบะฐะฝะพะผ (ั€ะฐะทะฝะพัั‚ัŒ ัั€ะตะดะฝะธั… hidden states ะดะพะผะตะฝะฝั‹ั… vs ะบะพะฝั‚ั€ะฐัั‚ะฝั‹ั… ะฟั€ะพะผะฟั‚ะพะฒ) ะธ ะฒัˆะธั‚ะพ steer-ะฟะฐั‚ั‡ะตะผ ะฒ ะฒั‹ั…ะพะดะฝั‹ะต ะผะฐั‚ั€ะธั†ั‹ (down_proj/o_proj/out_proj) ะฒั‹ะฑั€ะฐะฝะฝั‹ั… ัะปะพั‘ะฒ:

  • ะฑะฐะทะพะฒะฐั ะผะพะดะตะปัŒ: Qwen/Qwen3.5-0.8B
  • ั‚ะตะผะฐ (ะดะพะผะตะฝ): Python language & PEPs โ€” Python language, PEPs, coding conventions (EN)
  • ั€ะตะถะธะผ: steer, ฮฑ = 0.2
  • ัะปะพะธ: [15, 16, 17, 18, 20, 21, 22, 23]
  • ะณะตะนั‚ ัะบะฐะฝะฐ: AUC = 1.0 (PASS)

ะ ะพัƒั‚ะตั€ (ัะผะฑะตะดะดะธะฝะณะธ + margin) ะฒั‹ะฑะธั€ะฐะตั‚ ัั‚ัƒ ัะฑะพั€ะบัƒ ะฒะผะตัั‚ะพ ะฑะฐะทะพะฒะพะน ะดะปั ะทะฐะฟั€ะพัะพะฒ ั‚ะตะผั‹.

ะœะตั‚ะพะด

ะ’ะดะพั…ะฝะพะฒะปะตะฝะพ Weight-Space Programming (Takeyama et al., MIT CSAIL, 2026): ะฟั€ะฐะฒะบะธ ะฒะตัะพะฒ ะฑะตะท ะณั€ะฐะดะธะตะฝั‚ะพะฒ. ะ ะฐััˆะธั€ะตะฝะธะต ะปะฐะฑั‹ Ivanna: ะพะฑั€ะฐั‚ะฝะฐั ะพะฟะตั€ะฐั†ะธั โ€” ะฝะต ะฐะฑะปัั†ะธั ะฟะพะฒะตะดะตะฝะธั, ะฐ steer-ัƒัะธะปะตะฝะธะต ะดะพะผะตะฝะฝะพะณะพ ะฝะฐะฟั€ะฐะฒะปะตะฝะธั, ะฒัˆะธั‚ะพะต ะฒ ะฒะตัะฐ (ั€ะฐะฑะพั‚ะฐะตั‚ ั‚ะฐะผ, ะณะดะต ะธะฝั„ะตั€ะตะฝั-ั…ัƒะบะธ ะฝะตะดะพัั‚ัƒะฟะฝั‹).

Quantized GGUF

Q8_0 quantization lives in the sibling repo Qwen3.5-0.8B-Python-SoftMoE-GGUF.

Downloads last month
251
Safetensors
Model size
0.9B params
Tensor type
F32
ยท
BF16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for NikitaLazarev/Qwen3.5-0.8B-Python-SoftMoE

Finetuned
(989)
this model
Quantizations
1 model