Qwen3.5-0.8B (e2e pilot) โ€” Soft MoE expert ยซPython language & PEPs (RAG v2)ยป

EN

A soft MoE domain expert derived from a dense model without any training (weight-space steering): the domain direction dฬ‚ is extracted via an activation scan (mean hidden-state difference between domain and contrast prompts) and baked into the weights with a rank-one steer patch on the output matrices (down_proj/o_proj/out_proj) of the selected layers:

  • base model: Qwen/Qwen3.5-0.8B
  • theme (domain): Python language & PEPs (RAG v2) โ€” Domain prompts sampled from the platform RAG corpus: official Python docs EPUB + PEPs (30 chunks). V2 rebuild experiment.
  • patch mode: steer, alpha = 0.1
  • layers: [15, 16, 17, 18, 20, 21, 22, 23]
  • scan gate: AUC = 1.0 (PASS)

An embedding-space router (cosine margin) picks this build instead of the base model for requests matching the theme.

Method

Inspired by Weight-Space Programming (Takeyama et al., MIT CSAIL, 2026): weight edits without gradients. Ivanna Lab extension: the inverse operation โ€” instead of ablating a behavior, a domain direction is steer-amplified and baked into the weights (works where inference hooks are unavailable).

Created with Ivanna Lab โ€” self-hosted LLM model lab.

Notes

V2 rebuild: activation scan on 30 domain prompts sampled from the platform RAG corpus (official Python docs EPUB + PEPs), steer alpha=0.1, position=last. Benchmark: code2text_python (CodeXGLUE) BLEU-4 = 1.0534 on 150 samples โ€” best of the tested grid (vanilla base 1.0519, alpha=0.2 build 1.0289). Tested in Ivanna Lab: https://github.com/wiskeone/ivanna-lab

Benchmark results

Tested in Ivanna Lab (self-hosted LLM model lab, GitHub) with its built-in benchmarks module (lm-eval-harness). Task: code2text_python (CodeXGLUE), smoothed BLEU-4, 150 samples, greedy decode:

Benchmark chart

Build BLEU-4
This model (RAG-sampled prompts, alpha=0.1, position=last) 1.0534
alpha=0.1, weight_lab prompts, position=last 1.0525
vanilla base (Qwen3.5-0.8B) 1.0519
alpha=0.1, RAG prompts, position=mean 1.0436
alpha=0.2, weight_lab prompts, position=last 1.0289
alpha=0.2, weight_lab prompts, position=mean 1.0167

HumanEval (164, full) is identical for base and expert (0.2256) โ€” the steer patch is safe for general coding. Deltas on standard benches are small: the docs-direction targets style adherence, measured properly by domain-native benchmarks.

GGUF

qwen3.5-0.8b-python-softmoe-v2-q8_0.gguf โ€” Q8_0 quantization converted with llama.cpp (convert_hf_to_gguf.py). Runs in llama.cpp / LM Studio / Ollama directly โ€” the steer patch is baked into the weights.

RU

ะœัะณะบะธะน MoE-ัะบัะฟะตั€ั‚, ะฟะพะปัƒั‡ะตะฝะฝั‹ะน ะธะท ะฟะปะพั‚ะฝะพะน ะผะพะดะตะปะธ ะฑะตะท ะดะพะพะฑัƒั‡ะตะฝะธั (weight-space steering): ะฝะฐะฟั€ะฐะฒะปะตะฝะธะต ะดะพะผะตะฝะฐ dฬ‚ ะฒั‹ะดะตะปะตะฝะพ ะฐะบั‚ะธะฒะฐั†ะธะพะฝะฝั‹ะผ ัะบะฐะฝะพะผ (ั€ะฐะทะฝะพัั‚ัŒ ัั€ะตะดะฝะธั… hidden states ะดะพะผะตะฝะฝั‹ั… vs ะบะพะฝั‚ั€ะฐัั‚ะฝั‹ั… ะฟั€ะพะผะฟั‚ะพะฒ) ะธ ะฒัˆะธั‚ะพ steer-ะฟะฐั‚ั‡ะตะผ ะฒ ะฒั‹ั…ะพะดะฝั‹ะต ะผะฐั‚ั€ะธั†ั‹ (down_proj/o_proj/out_proj) ะฒั‹ะฑั€ะฐะฝะฝั‹ั… ัะปะพั‘ะฒ:

  • ะฑะฐะทะพะฒะฐั ะผะพะดะตะปัŒ: Qwen/Qwen3.5-0.8B
  • ั‚ะตะผะฐ (ะดะพะผะตะฝ): Python language & PEPs (RAG v2) โ€” Domain prompts sampled from the platform RAG corpus: official Python docs EPUB + PEPs (30 chunks). V2 rebuild experiment.
  • ั€ะตะถะธะผ: steer, ฮฑ = 0.1
  • ัะปะพะธ: [15, 16, 17, 18, 20, 21, 22, 23]
  • ะณะตะนั‚ ัะบะฐะฝะฐ: AUC = 1.0 (PASS)

ะ ะพัƒั‚ะตั€ (ัะผะฑะตะดะดะธะฝะณะธ + margin) ะฒั‹ะฑะธั€ะฐะตั‚ ัั‚ัƒ ัะฑะพั€ะบัƒ ะฒะผะตัั‚ะพ ะฑะฐะทะพะฒะพะน ะดะปั ะทะฐะฟั€ะพัะพะฒ ั‚ะตะผั‹.

ะœะตั‚ะพะด

ะ’ะดะพั…ะฝะพะฒะปะตะฝะพ Weight-Space Programming (Takeyama et al., MIT CSAIL, 2026): ะฟั€ะฐะฒะบะธ ะฒะตัะพะฒ ะฑะตะท ะณั€ะฐะดะธะตะฝั‚ะพะฒ. ะ ะฐััˆะธั€ะตะฝะธะต ะปะฐะฑั‹ Ivanna: ะพะฑั€ะฐั‚ะฝะฐั ะพะฟะตั€ะฐั†ะธั โ€” ะฝะต ะฐะฑะปัั†ะธั ะฟะพะฒะตะดะตะฝะธั, ะฐ steer-ัƒัะธะปะตะฝะธะต ะดะพะผะตะฝะฝะพะณะพ ะฝะฐะฟั€ะฐะฒะปะตะฝะธั, ะฒัˆะธั‚ะพะต ะฒ ะฒะตัะฐ (ั€ะฐะฑะพั‚ะฐะตั‚ ั‚ะฐะผ, ะณะดะต ะธะฝั„ะตั€ะตะฝั-ั…ัƒะบะธ ะฝะตะดะพัั‚ัƒะฟะฝั‹).

Downloads last month
125
Safetensors
Model size
0.9B params
Tensor type
F32
ยท
BF16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for NikitaLazarev/Qwen3.5-0.8B-Python-SoftMoE-v2

Quantized
(436)
this model